Jason Levitt has been teasing me in our discussions on cross-domain requests about Yahoo's upcoming authentication API.
The recurring problem: how to offer web APIs that can be mashed up but involve personal data?
You want to allow for a large number of third parties to integrate with your services, but don't want phishing sites to abuse them.

Let me do a quick re-cap of the problem space before analyzing the pieces of Yahoo's solution.

Communication techniques:

• cross-domain web APIs using script includes,
• accessing web APIs accross domains using a web proxy,
• using a client-based cross-domain solution (see FlashXMLHttpRequest),
• neutered XMLHttpRequest with cross-domain support (see Chris Holland's ContextAgnosticXMLHttpRequest and Douglas Crockford's JSONRequest),
• cross-document messaging across domains (see CrossDocumentMessageEvent from the WHATWG).

In all these cases, there is no good authorization story, that would allow for working with personal data stored in the service in a secured way.

Authorization techniques:

The problem is that API keys and crossdomain policy files are too restrictive because the service needs to decide which third-parties to let in.

On the other end, access control based on the user authentication cookies are very open to un-planned integration, but also create a huge phishing risk.
This is a classic example of the confused deputy problems that appear in principal-based security models.

As a result, most web APIs today don't involve any user data (search, maps, ...) or non sensitive user data.

Yahoo APIs:

Let me re-iterate that I don't think this protocol is about Identity, unlike Passport, TypeKey or CardSpace (aka. InfoCard), but rather simply authority and access. This characteristic is important: we want services to cooperate without being tighly coupled at the identity level. Drew Dean's slides frames the issue as allowing 'Pseudonymous delegation of partial rights', which means the names of a user in different services don't have to match and the authority that is granted is granular.

What's great about this model is that the authority carried by a capability can be as granular as the design and scenario require, and are only be given out to third-parties under certain conditions, which again are chosen to fit the desired requirements and user experience.

For example, the authority granted could vary in range in action and scope: a handle could give access to the user's entire data, or maybe only partial access to part of the user's data.
The design of the capabilities could also comprise additional dimensions, such as a time restriction. For example, a capability could be only valid for 24 hours.
One of the myths of capability systems is that capabilities cannot be revoked. It is actually possible and in Yahoo's design, any granted authority can be revoked by the user at any time.

One common policy for giving out capabilities is to get consent from the user. The screenshots of the F-Spot integration with Flickr (found on this thread) show the Yahoo consent UI.
Although I don't like the desktop/web integration in this scenario and I have some concerns about repeatedly prompting the user for consent, I believe that this approach has a lot of potential for cross-domain service integrations on the web.
Cross-domain support in browsers will be the main remaining link missing to unleash some really cool web apps. In the meanwhile, you can use FlashXMLHttpRequest or some other cross-domain workaround.

I look forward to reading the documentation when the protocol is released and trying out the resulting user experience in practical scenarios. Let me know if you find any other information.
Jason mentioned that the protocol is open and can be simply implemented, which means that it could be supported by other services and hopefully used in a wide variety of mashups.

I've been working with Jason Levitt (from Yahoo) recently to iron the wrinkles out of my Flash-based technique for making cross-domain requests. The main focus was to extend the reach of the technique to a larger audience, by adding support for Flash 7, which is more widely deployed than Flash 8.

Try the new FlashXMLHttpRequest demo.

Brad Neuberg's awesome work on DojoExternalInterface gave us a major boost. It emulates the most recent javascript to flash communication API (ExternalInterface, in Flash 8) accross platforms including Flash 7. The library hides differences with browsers and flash versions as much as possible, and works around various performance limitations.

Thanks again to Jason for his interest, feedback and support. I'm pretty excited to see what cool stuff he'll cook using this and the new web APIs from Yahoo.


If you only want to deploy this component, you can download the 'binary' package.
To modify the flash source files and re-build them, get the 'source' package.

As you can see in the demo/index.html file, after including dojo.js and FlashXMLHttpRequest.js, you'll need to initialize dojo and the flash object by calling InitFlash with the name of a function. That function will be invoked once the flash object is loaded and ready to make requests.
From there on, you can create FlashXMLHttpRequest instances and use the 'open', 'onload' and 'send' methods almost as you would with a regular XMLHttpRequest object. You can also call 'setRequestHeader', but only to set the content type request header.

More generally, FlashXMLHttpRequest still has some limitations, due to the native Flash capabilities.
First, access to other domains is restricted by use of a crossdomain.xml file.
Second, you can only make GET and POST requests. It will become possible to support other verbs, such as PUT, DELETE or HEAD, with the new APIs provided by Flash 8.5.

Let me know if FlashXMLHttpRequest is useful to you and if you encounter any bugs. Here's a more convenient permalink to this post: http://blog.monstuff.com/FlashXMLHttpRequest.

Knowledge is power. Yet, it is a power that is not trivial to maintain. The organizational body of knowledge is always growing, since new people are joining the enterprise, more projects are being developed and more experience in the problem domain and the relevant technology is gained. However, the fact that the sum of knowledge in the organization is ever growing, does not guarantee that this vast amount of knowledge will indeed be used when and where needed. On the contrary, in some cases, the more knowledge there is in the organization, the less accessible it is to people who really need it, when they need it. Thus, the main concern in this context is not how to gain more knowledge, but how to maintain it: how to make it accessible in such a manner that will make it useful.

Read the article...

Ludema & Boyink Sod Farm is located 1/2 hour east of Grand Rapids, MI in Clarksville, MI and provides residential and commerical sod sales and delivery to all of West Michigan. You can also pick it up yourself at our farm.http://www.LudemaBoyinkSod.com/

Upgrading to Mandrake 10

Last night I decided to upgrade my home Linux box, which was running Mandrake 9.2, to the newly-released Mandrake 10.

I run Mandrake at home, and until recently, on my work machine, too.  Now at work I run Red Hat 9, but only because we use ClearCase (don't get me started) as our version control software, and it patches into the kernel, and only supports Red Hat.  So I was forced (kicking and screaming) to switch to Red Hat at work.

Now, this is just my personal opinion, and I'm sure I'll get flak for it, but Mandrake beats Red Hat for ease-of-installation and use.  Not by a whole lot, but enough that I perfer Mandrake.  My upgrade to Mandrake 10 is a good example of what I'm talking about.

Mandrake has not yet released the ISO files of the 'official' version of Mandrake 10, as they want their preferred (read that, paying) customers to be able to get it first.  But you can get 'official' Mandrake 10 if you are willing to follow a few simple steps.

• First, find a Mandrake mirror site.  I used raven.cslab.vt.edu. 
• Using any modern browser that lets you access an ftp site, go to the directory containing the official Mandrake 10 release, which on that site was '/pub/linux/mandrakelinux/official/10.0/i586'.  Then go into the 'images' subdirectory.
• Download 'boot.iso', and burn it to a CD.
• Using your existing Mandrake 9.2 installation, write down the following information: system name, system domain, system IP address, DNS IP address, gateway IP address (if you use a gateway to access the internet)
• Insert the CD you just burned, and reboot your system so that it boots from the CD
• A Mandrake 10 special boot loader will come up.  It will ask you for the info I had you write down, including the location of the ftp site and the directory for the Mandrake 10 source.  Use the raven site from above.
• If you have broadband, it will spend almost 3 hours downloading Mandrake 10.  If you are on a dial-up, this may take all night.  Fire up your Windoze game machine and play 'Splinter Cell' for a while.
• Once the download is done, the installer will walk you through the normal installation steps.  Tell it you are upgrading an existing system.  It will recognize all of your current drives, partitions, setting, everything.
• Done.  You now have a Mandrake 10 system.

Once Mandrake does the wide release of Mandrake 10, installation will be even easier.  Find a site hosting the ISO files, download them using Bittorrent, burn the CDs, reboot from CD1, and off you go.  But the fact that I can do a full upgrade over the Internet using only the steps I gave above, with no problems what so ever... well, bravo to Mandrake.

Oh, and Mandrake 10 comes with Python 2.3.3.

Working With Fixed Record Length CSV Files

Yesterday I wrote about working with fixed record length files, and presented a class for making it easy.  Today I'll extend that idea to handle records containing csv data.

The new CSVRecordFile class inherits from the RecordFile class, overriding the read and write methods to add csv parsing and formatting.  This allows you to read and write fixed length csv data records in random order.  The StringCSVAdaptor class (presented earlier) is used to enable us to use Python2.3's csv module with strings.  This is necessary since the csv modules reader and writer functions expect to work with interables, such as a file-like object or a sequence.

Matt Goodall took me up on the need for the StringCSVAdaptor class, rightly pointing out a simpler way of handling the problem (thanks, Matt!  I think you are reader #4 of this weblog, and the first person to leave me a comment!)  Sadly, Matt's suggestion does not fit with the problem domain I'm using CSVRecordFile for.  For one thing, I need both a csv reader and writer.  I only want to create these objects once per CSVRecordFile instance, and then use them to parse/format many records in random (not sequential) order.  Matt's solution, while useful for simple one-shot csv needs, looks to me to require the creation of the reader and writer for each record that is to be parsed (because he wraps the string to be parsed in a list to make it an iterable).  You can read Matt's comments from yesterday's post.

Put this code in a file called csvrecfile.py:

"'This file contains the CSVRecordFile class, for working with fixed length
record files, where the records contain csv data."'

__author__ = 'Mike Kent'
__version__ = '$Id$'.split()[-2:][0]

import recfile
import csv
import csvadaptor

class CSVRecordFileFmtError(Exception): pass

class CSVRecordFile(recfile.RecordFile):
    "'This class provides a standard way to handle files which are layed out
    as fixed-length records containing csv data, where each record is padded 
    to its proper length with a padding character, and may be optionally 
    terminated with a record terminator string."'
    def __init__(self, filename, mode, reclen, recpad=', recterm=None,
            **csvKwParams):
        recfile.RecordFile.__init__(self, filename, mode, reclen, recpad,
            recterm)
        
        self.csvAdaptor = csvadaptor.StringCSVAdaptor()
        self.csvReader = csv.reader(self.csvAdaptor, **csvKwParams)
        self.csvWriter = csv.writer(self.csvAdaptor, **csvKwParams)
        return

    def read(self, recNum):
        "'Read a record containing csv data by record number, and return a
        list of strings.
        Record numbers start a 1.  An empty list will returned on end of
        file."'
        self.csvAdaptor.data = recfile.RecordFile.read(self, recNum)
        try:
            rec = self.csvReader.next()
        except csv.Error:
            raise CSVRecordFileFmtError

        return rec

    def write(self, recNum, valueList):
        "'Write a list of mixed-type values to a record, in csv format,
        by record number.
        Record numbers start with 1.  The record will be
        padded to the correct length using the padding character, and
        optionally terminated by the record terminator string.
        You can seek to, and write, records beyond EOF.  However, to
        append a new record to the current actual EOF, give a record number
        of 0.  This function returns the actual record number written to."'
        try:
            self.csvWriter.writerow(valueList)
        except csv.Error:
            raise CSVRecordFileFmtError

        return recfile.RecordFile.write(self, recNum, self.csvAdaptor.data)

Here are the unit tests.  Put this code in a file called test_csvrecfile.py:

#! /usr/bin/env python

import sys
import unittest
import csv
import csvrecfile

class TestCases_01_RecordFile(unittest.TestCase):
    def test_01_instantiate(self):
        recFileObj = csvrecfile.CSVRecordFile('test.txt', 'w+b', 20, ' ',
            'r', lineterminator=', quoting=csv.QUOTE_NONNUMERIC)
        self.assert_(recFileObj is not None)
        return

class TestCases_02_RecordFileWriteAdd(unittest.TestCase):
    def test_01_writeAddOne(self):
        recFileObj = csvrecfile.CSVRecordFile('test.txt', 'w+b', 20, ' ',
            'r', lineterminator=', quoting=csv.QUOTE_NONNUMERIC)
        self.assert_(recFileObj is not None)
        rec = [ 1, 2, 'three' ]
        recFileObj.write(0, rec)
        recFileObj.flush()
        testFileObj = file('test.txt', 'rb')
        expected = '1,2,'three'       r'
        newRec = testFileObj.read(20)
        self.assertEqual(newRec, expected)
        return
        
    def test_02_writeAddSeveral(self):
        recFileObj = csvrecfile.CSVRecordFile('test.txt', 'w+b', 20, ' ',
            'r', lineterminator=', quoting=csv.QUOTE_NONNUMERIC)
        self.assert_(recFileObj is not None)

        for count in range(5):        
            rec = [ 1, 2, 'Record %d' % (count + 1) ]
            recFileObj.write(0, rec)
            
        recFileObj.flush()
        testFileObj = file('test.txt', 'rb')

        for count in range(5):
            expected = '1,2,'Record %d'    r' % (count + 1)
            newRec = testFileObj.read(20)
            self.assertEqual(newRec, expected)
        return

class TestCases_03_RecordFileWriteRandom(unittest.TestCase):
    def setUp(self):
        recFileObj = csvrecfile.CSVRecordFile('test.txt', 'w+b', 20, ' ',
            'r', lineterminator=', quoting=csv.QUOTE_NONNUMERIC)
        self.assert_(recFileObj is not None)

        for count in range(5):        
            rec = [ 1, 2, 'Record %d' % (count + 1) ]
            recFileObj.write(count + 1, rec)

        recFileObj.close()
        return

    def test_01_writeRandomOne(self):
        recFileObj = csvrecfile.CSVRecordFile('test.txt', 'r+b', 20, ' ',
            'r', lineterminator=', quoting=csv.QUOTE_NONNUMERIC)
        self.assert_(recFileObj is not None)

        recNum = 2
        rec = [ 3, 4, 'New record %d' % recNum ]
        recFileObj.write(recNum, rec)
        recFileObj.flush()
        testFileObj = file('test.txt', 'rb')

        expected = '3,4,'New record %d'r' % recNum
        testFileObj.seek(20)
        newRec = testFileObj.read(20)
        self.assertEqual(newRec, expected)
        return

    def test_02_writeRandomSeveral(self):
        recFileObj = csvrecfile.CSVRecordFile('test.txt', 'r+b', 20, ' ',
            'r', lineterminator=', quoting=csv.QUOTE_NONNUMERIC)
        self.assert_(recFileObj is not None)

        recNumList = [ 1, 5, 3, 2, 4 ]
        for recNum in recNumList:
            rec = [ 1, 2, 'New record %d' % recNum ]
            recFileObj.write(recNum, rec)
        
        recFileObj.flush()
        testFileObj = file('test.txt', 'rb')

        for count in range(5):        
            expected = '1,2,'New record %d'r' % recNumList[count]
            testFileObj.seek((recNumList[count] - 1) * 20)
            newRec = testFileObj.read(20)
            self.assertEqual(newRec, expected)
        return

class TestCases_04_RecordFileReadRandom(unittest.TestCase):
    def setUp(self):
        recFileObj = csvrecfile.CSVRecordFile('test.txt', 'w+b', 20, ' ',
            'r', lineterminator=', quoting=csv.QUOTE_NONNUMERIC)
        self.assert_(recFileObj is not None)

        for count in range(5):        
            rec = [ 1, 2, 'Record %d' % (count + 1) ]
            recFileObj.write(count + 1, rec)

        recFileObj.close()
        return

    def test_01_readRandomOne(self):
        recFileObj = csvrecfile.CSVRecordFile('test.txt', 'r+b', 20, ' ',
            'r', lineterminator=', quoting=csv.QUOTE_NONNUMERIC)

        rec = recFileObj.read(3)
        expected = [ '1', '2', 'Record 3' ]
        self.assertEqual(rec, expected)
        return

    def test_02_readRandomSeveral(self):
        recFileObj = csvrecfile.CSVRecordFile('test.txt', 'r+b', 20, ' ',
            'r', lineterminator=', quoting=csv.QUOTE_NONNUMERIC)

        recNumList = [ 1, 5, 3, 1, 2, 2, 4 ]
        for recNum in recNumList:        
            rec = recFileObj.read(recNum)
            expected = [ '1', '2', 'Record %d' % recNum ]
            self.assertEqual(rec, expected)
        return

if __name__ == '__main__':
    unittest.main()
    sys.exit(0)
    

Project Distributor: Introduction to our distributed web service model
So Darren and I have put in about a month now on the Project Distributor website. We are starting to reach that critical point where the site is pretty cool, we have plenty of users, we are thinking about running out of the allowable bandwidth for the demo site, and all sorts of other things that tend to happen all at once. Now, there are some problems you can design yourself out of, and others that you really have to throw some money at. Our latest enhancements can be summed up in a short list.

• Buy a domain name and start hosting in two places. Project Distributor.com should be up fairly soon to accompany MarkItUp.ASPXConnection.com
• Have people host their own versions of the application. And that means a big source release is in the future. At this juncture risk fragmentation.
• Design away fragmentation with a series of ingenious features that will make everyone want to use the application at hand.

I'm here to talk about the last two, since Darren already bought some additional hosting for us. The concept will be to release a fairly stable version of the application so that groups can host tools, code snippets and other source/binary releases for their teams to share. The application is very lightweight and easy to set-up, so it won't require a bunch of hand holding and configuration to get up and running initially. From our standpoint we solve a number of issues at this juncture. The most obvious problem is what we classify the Lutz Roeder use case. .NET Reflector is the key type of application we'd love to get hosted because it makes it a bit easier to find, not that Google does a bad job, we'd just like to get a bunch of tools in one place, with some features for feedback, new releases, and some cool client tools for publishing.

Now, Lutz would put his application up and he'd whack our bandwidth. He is the prime example of someone that should be hosting their own tools, but possibly using our interface. He doesn't have to, we haven't even asked him yet in fact, but if he decides to do so, then all the better for the web application moving forward. Users such as Lutz probably want a certain level of control over their own sites as well in terms of branding and controlling access. This will only come from hosting the application yourself (and maybe some other features we'll see later).

From a security standpoint many teams will also want to host their own servers. In this manner they get control over the hardware their sources and binaries are stored on. They can accept tools up to any maximum (instead of our imposed limits) and provide unlimited download bandwidth if they choose. Or they can take advantage of our gating mechanisms to make sure their server doesn't get overloaded with downloads and open their tools up to the public.

The only major problem from this source release is that the initial problem we were trying to solve, promoting the visibility of tools, starts to erode. You see, the more sites that host their own tools the harder it is to find the right site with the right tools. We are trying to solve this in a number of ways. The first is allowing users of a site to store bookmarks to other projects and external resources. This is only a temporary fix, because it still doesn't allow a mass search and categorization infrastructure required to truly promote the visibility of the tools being hosted. We have to come up with a solution that brings all of the sites, but we don't want to create just another portal or gateway site. That is boring. Now you have the background, so how will we solve the fragmentation issue?

Designing away Fragmentation
I won't lie to you, I've implemented this model several times, but have never had a project that was capable of really showing off the feature set we are about to talk about. The concept is to unify all of the sites, by allowing them to easily manage views of data from all of the sites combined. Each site owns their own content, maintains their own users, but in turn peers with other sites to obtain additional content.

Web services provide a dual feature set in this model. At the current level they allow us to generate really great client-side tools for managing, well, your tools! We have a drop-client target right now so you can drag and drop new releases to existing projects in just a few seconds. Some new tools for working with build systems to promote the source code up to the server are in the works. We natively integrate with your RSS reader and will have our own alert services in the drop client just in case you don't have one. There aren't any search or local caching features, but those are also planned for the drop client so you can background download new releases, just like Windows Update.

That doesn't solve fragmentation though, that just makes me realize how much work I have left to do. The second feature of web services lies in the ability for each site to aggregate data from the many other sites that are out there hosting the application. Remember, everything we make available at the service layer can also now be remoted. The more caching we put into the data layer, the more performant the entire process will be, and we can even tune the caching depending on whether the data layer is merging off-site contents or database contents.

Peer Sites
I'm sure there is another name out there somewhere, but for the past 2 years I've called these peer sites. Each instance of the project distributor will have a number of options allowing for adding peers that will be aggregated and added to the local collection while users traverse the site. The first step is to get the peer sites running in a read-only mode. And set up some really great options so the entire process can be controlled. This solves a number of use case scenarios for us including the following.

• Fragmentation can be mitigated through proper configuration. If everyone aggregates 5 or 6 sites into their peers, then we have a huge network now of interconnected peers and users can pick and choose which one they use for purposes of searching the tool network.
• Peer connections are unidirectional or bidirectional. Access is configurable. Teams can include tools from external sites while keeping their own tools completely private. They can exist behind a DMZ or a private network.
• Users can host their own personal tool sites in the same manner as the team sites. They can configure statically which projects to make available even. In this way you can build a collection of personal tools that you love, and have the latest information automatically update on your machine for your perusal.

Peer sites solve plenty of visibility issues, but that is pretty much all they solve for now. We still want to enable all of the features available to the client tools. After all, the web service methods and proxy infrastructure is in place to do so much more.

Master Sites
Well, we want to solve another problem. That is where you edit your data. A master site is where the users, groups, projects, etc... are all hosted, but thankfully, you'll be able to log in through any site (assuming it is peered with your master site) and then edit your own projects and such. This is a remote principal context and is actually one of the cooler features associated with the peering functionality of project distributor. We'll be fully secure in our login and credentials region, but unfortunately we'll still be transferring data in open text in the short term. Maybe we'll fix that with enough push back.

Clone Sites
A clone site is where we empower a site to act on behalf of a master site. For me, my local project distributor is currently cloned to the main project distributor site. What does this mean? Right now it means I get all of the data from PD, and that users who trust my site can log-in to their project distributor accounts and cross edit data. Pretty nice if you ask me. It basically means you can fully host a project distributor installation and never, ever have to install a database server. Users can just act on behalf of a remote server.

Configuration
This isn't a super reusable model like some of those you read about in the popular software architecture books, and it probably accounts for why master/peer/clone sites don't exist very often. The considerations for every option are heavily customized to the problem being solved, and I'm sure we'll be making modifications or updating the configuration context for a while. Right now you can independently configure your primary server type, whether master or clone, whether or not users can use you for a pass-through authentication and edit server, whether or not web services are enabled so peers can enable unidirectional only communications, setting up asymmetric security credentials. Man, you name it and it is in there

For the peer section we have full and selective modes. A full peer pulls all of the data on the remote peer locally for display (in a delay caching manner, just like you'd expect, unless you set up a scheduled pull which is also possible). I expect most people to configure full peers because they really are really easy to set up and maintain. A selective peer is where you specify the groups/projects that you want to display. This is best for a user setting up their own personal toolbox who wants to select a couple of items from many different peers.

We have an extensively exhaustive configuration module already and we'll be continuously adding more to it. The concept is to easily modify your toolbox to your own designs without having to touch the code. If we haven't given you enough options to satisfy your need then we'll have to make something up, because I'm just about running out ;-)

These are the basics of the model ideas I have for project distributor. That doesn't mean Darren doesn't have other great ideas happening as well. He has some pretty extensive UI enhancements, but I'll let him talk about those. We even have another product idea that is kind of a bolt-on for project distributor, but that is probably a couple of months out putting it into next year. Unfortunately we have too many ideas for our own good right now. Better than not having any ideas I guess. I'll try to drop some code with some of the ideas above, that way you can get a look at how the entire system is implemented. I have some diagrams as well, but I'm far too tired right now to add the img tags to the HTML view.