Wednesday, August 02, 2006

Smarter Multithreading: AJAX in Windows Forms and Software Transactional Memory

Whether juggling CPU- and IO-bound functions on the server, or doing heavy lifting on a client without blocking the UI thread, handing jobs off to thread pools is a fact of life for today’s programmers. In this post I examine the basic plumbing we’re using to get the job done, then compare it to the programming patterns AJAX is making all the cool kids learn.

.NET supplies its own very easy to use ThreadPool, but that class suffers from a few drawbacks. We’ve started to use Ami Bar’s Smart Thread Pool for our programming and are very pleased with it. Of most interest to us, it makes capturing thrown Exceptions simple and uniform. The Smart Thread Pool sits at the heart of our smart client UI. This UI does some very heavy lifting: corresponding with web services, uploading/downloading large files, and speaking with recalcitrant DCOM servers. We wanted a clean way for people to write code to handle these problems without blocking the UI thread. We settled on an explicit Controller/View pattern.

When our main form starts up, it creates a controller object. (Some names have been changed to protect the innocent).

public ThingForm() { ... _controller = new ThingController(this); }

The controller handles all UI events asynchronously. For instance a button click does something like

_controller.AsyncInvoke(new WorkItemCallback(_controller.ScanForMxdSessions));

The AsyncInvoke function on the controller uses the Smart Thread Pool to spin up the task on another thread. This leaves the UI thread free to continue updating and responding to events.

ClientThreadPool.Pool.QueueWorkItem( function, this, new PostExecuteWorkItemCallback (HandleThreadPoolException));

This is where the smart thread pool comes in handy — I have a single function which can handle unexpected exceptions for all controller operations. This can do logging, put up a message box, etc.

This exception handler and other controller functions do need to eventually update the form UI. They use a sister function which invokes a delegate on the form’s STA thread.

StaInvoke(new StaThunk(Sta_UpdateComponentTree));

StaInvoke simply calls the parent Form’s Invoke function, which queues the delegate up to run on the parent STA thread. Then the Sta_XXX functions are free to pretend they’re single threaded (because they are being executed on the STA thread) and poke at the UI components all they want (in the case above, binding a tree view). We know they’ll execute quickly because they are only examining model state and making trivial updates to UI components.

This paradigm enables an extremely responsive user interface at the same time it allows execution of long-running queries. Sound familiar? It’s the AJAX pattern all the cool kids are learning, but in Windows Forms. In AJAX, tasks are always kicked off asynchronously, whether explicitly (e.g. XmlHttpRequest request or setTimeout), or implicitly (event handlers). The UI writer must present an interface that can continuously be updated by background worker tasks. Yahoo Maps beta is one of my favorite current examples of this.

JavaScript provides a lot of training wheels: all threading is hidden from you. But in .NET, you have to deal with it yourself. The pattern raises some interesting thread safety issues. If the user kicks off two instances of the same job, how can we be sure the data is presented correctly? In my example above, I am scanning for “MxdSessions”, and then updating a tree control with the results. Someone could press the button twice, and it’s utterly unknown who will get to the tree control first. Even worse, how do we keep from creating corrupted results, e.g. a list of MxdSession half from one request, and half from another?

We’re still looking at alternatives, but the keep-it-simple approach is working. Since the real work done on the “model” is multithreaded and doesn’t have to worry about blocking UI updates, we simply use explicit mutexes around certain controller functions. For instance, all functions that would be sensitive to “MxdSessions”, have a lock(_syncMxdScan) around the important bits. Sure, the user can press the button three times, but the lock will force those scans to happen serially. On the other hand, a user request to do something unrelated (like logging into a remote server), doesn’t need to block on any of this.

Then we get back to those UI-updating functions. They are freed from the need to synchronize around form data structures, because they are executing on an STA thread. However they still need to worry about the reliability of the data they’re reading from the controller — it could change while they’re looking at it. Using the same mutex that the controller functions use, they attempt to pull a consistent chunk of data out of the controller.

NameAndId[] orgs = null; Study[] studies = null; lock(_syncContextUpdate) { orgs = _ownerOrgs; studies = _studies;}

This function can now use orgs and studies at will, knowing they are working with a copy of the data that will remain consistent during their execution, because all controller functions which could change them also lock on _syncContextUpdate.

It makes one yearn to simply treat the model’s in-memory state as a database. A modern RDBMS automatically takes care of ensuring you have a consistent view of data during your own transaction. I’m convinced that Software Transactional Memory libraries will go mainstream very soon, enabling this much easier programming model.

In the meantime, the above ramblings lead me to be a bit more dubious of JavaScript’s training wheels. If I am updating model state at the same time someone else is rendering it, how can I be sure of consistency? Despite a great deal of searching, I still don’t know how many threads the JavaScript interpreters in major browsers are using. And even if I did, there’s no such thing as a mutex in JavaScript. In another post I’ll examine how Software Transactional Memory might be simulated in JavaScript and .NET.

Technorati Tags:

Pander to me, TiVo!

The wireless adapter for my TiVo arrived today (the day after I ordered it with "3-5 business days" shipping. Despite the insult of having to buy theirs instead of one off the shelf (they don't support most of the new adapters out there), it's really nice not to have a telephone cord going from room to room.

I've commanded TiVo to fill itself with the finest products our civilization has to offer: The Daily Show, Battlestar Galactica, and Boston Legal for me; Seinfeld and Sex & the City for Amy. I'm reminded of the Calvin & Hobbes comic showing Calving sitting in front of his TV with a big grin. &;quot;Pander to me!" he says.

It that were subjunctive, I wouldn't know it

So apparently the title of this entry is in the subjunctive. Amy's learning about this construction, because it's common in French. In English, it seems a random categorization only invented by grammar geeks.

"The subjunctive mood (sometimes referred to as the conjunctive mood) is a grammatical mood of the verb that expresses wishes, commands (in subordinate clauses), emotion, possibility, judgment, necessity and statements that are contrary to fact."

Right. I can't contest the fact that learning the grammar of a highly regular language (like a Latin one) helps you up the learning curve. But isn't it a bit silly to look for this stuff in English? God, we barely conjugate our verbs anymore.

Monday, July 31, 2006

Let slip the bugs of war!

It should be a fun weekend. I ordered several thousand ladybugs and a few million nematodes from ARBICO, an organic pest control mail order house. Why they don't just sell a few hundred ladybugs at a time is a mystery to me. But anyway, it's the battle of the bulge in my backyard right now and I need reinforcements.

My plants seemed healthier when I never mowed the lawn. Now that a service does it, I think I have more infestations in my garden. Perhaps its a coincidence, but I think the nice low grass and lack of weeds take away the hiding places a lot of the beneficial insects used to live in. Now all that can thrive are aphids, ants, and fungus. And who wants those?

Cry Havoc! And let slip the bugs of war!

Sunday, July 30, 2006

API Exegesis (or, how to use IBasicGeoprocessor's useSelectedInput argument)

So I'm programming away, innocently trusting the reference material for ESRI ArcObject's IBasicGeoprocessor.Intersect function. The function claims it will intersect two feature classes, and honor a selection set on them. Because our big ESRI application stores data in extremely large SDE feature classes (millions of features) but typically does intersect operations on only a few hundred at a time, we have had to resort to tricks to get intersection performance acceptable. Typically, we script an export of appropriate data to a local geodatabase and perform intersections there. This is unfortunate, because it involves a lot of disk traffic for something that could theoretically be pushed down to SDE, or at least done with the right feature cursors.

It was this afternoon that I noticed the useSelectedInput and useSelectedOverlay arguments to the Intersect function. Selection of the appropriate subset is typically quite fast, so perhaps this was the answer I was looking for. The arguments are boolean, which begs the question: which selection set will they use?

The official developer help notes "The useSelectedInput state refers to whether or not a selected subset of the InputTable is to be intersected. True signifies that a selected subset will be intersected. False signifies that intersect operation will ignore any selected subset.". It's a boolean, so unless ESRI is willing to magically reach into your mind and figure out where on the local stack you stored your favorite ISelectionSet pointer, this won't do you any good.

This is when I noticed a funny disconnect between the official description of the argument, and the commentary for the interface as a whole. Like monks in the middle ages copying manuscripts, I wondered who was right? Was Mary a virgin or a young woman?. The summary description said "[i]ntersect clips the features of a line or polygon layer". So who was right? Did the thing intersect feature classes (abstract collections of shapes) or layers (pieces of an actual map).

In the name of abstraction, the arguments to specify the feature classes to be intersected were ITable. Not only is that no help, but it's silly: ITables don't even have shapes. It should be IFeatureClass or IFeatureLayer. It turns out you have to pass the function an IFeatureLayer because those carry implicit selection sets. If you pass the function an IFeatureClass it will work, but there is no way to specify a selection set.

This is wrong on many levels. They should have:

  • Provided two overloads of the function. One with IFeatureClass and ISelectionSet arguments, and one with IFeatureLayer and boolean arguments.
  • Or, raised a meaningful error message if the function was called with an IFeatureClass and true.

As it turns out, it was all for nought. Even though I was able to successfully talk the code into honoring my selection sets, the intersect was too slow. It appears that all it does is do the intersection first, without the selection sets, then filters the result. In this case, it tries to intersect about 400,000 polygons with 280,000 others. That's probably slower than intersecting 288 with 14.

Once again, ESRI, you get points for being brave enough to expose your APIs in such detail. But this is just plain bad who-let-the-intern-write-the-function programming. Better luck next time.

For the record, here's how you do it.

public bool IntersectPairNoExport( string sourceFeatureClass, string sourceWhere, AoWorkspaceContext overlayCtx, string overlayFeatureClass, string overlayWhere, bool constrainOverlayBySourceSelectionExtent, esriSearchOrder overlaySearchOrder, AoWorkspaceContext intersectCtx, string intersectFeatureClass) { IFeatureClass sourceFc = null, overlayFc = null, intersectFc = null; IFeatureLayer sourceLayer = null, overlayLayer = null; IFeatureClassName intersectName = null; IQueryFilter sourceQf = null, overlayQf = null; ISelectionSet sourceSel = null, overlaySel = null; IEnumGeometryBind enumGeometryBind = null; IGeometryFactory geometryFactory = null; IGeometry sourceGeometries = null; IBasicGeoprocessor gp = null; try { // the geoprocessor claims to honor selected subsets. however the only way this works // is if you pass it feature layers instead of plain tables. feature layers have an implicit // selection set. how lame is that. In cases where the sets to be intersected are very // small, this saves us the trouble of copying to a temporary feature class. However, it // seems to do nothing at all against very large datasets -- it is still necessary to // pull out selected subsets into smaller datasets and intersect them independently. sourceFc = OpenFeatureClass(sourceFeatureClass); sourceQf = new QueryFilter(); sourceQf.WhereClause = sourceWhere; sourceLayer = new FeatureLayerClass(); sourceLayer.FeatureClass = sourceFc; ((IFeatureSelection) sourceLayer).SelectFeatures(sourceQf, esriSelectionResultEnum.esriSelectionResultNew, false);// esriSelectionType.esriSelectionTypeHybrid, esriSelectionOption.esriSelectionOptionNormal, Workspace); sourceSel = ((IFeatureSelection) sourceLayer).SelectionSet; if (sourceSel.Count == 0) return false; // we want to limit the selection by the envelope of the source fc // see http://forums.esri.com/Thread.asp?c=93&f=992&t=90612&mc=10 for this fun technique overlayFc = OpenFeatureClass(overlayFeatureClass); if (constrainOverlayBySourceSelectionExtent) { enumGeometryBind = new EnumFeatureGeometryClass(); enumGeometryBind.BindGeometrySource(sourceQf, sourceSel); geometryFactory = new GeometryEnvironmentClass(); sourceGeometries = geometryFactory.CreateGeometryFromEnumerator(enumGeometryBind as IEnumGeometry); overlayQf = NewQueryFilter((ITable)overlayFc, overlayWhere, sourceGeometries.Envelope, overlaySearchOrder); } else { overlayQf = new QueryFilter(); overlayQf.WhereClause = overlayWhere; } overlayLayer = new FeatureLayerClass(); overlayLayer.FeatureClass = overlayFc; ((IFeatureSelection) overlayLayer).SelectFeatures(overlayQf, esriSelectionResultEnum.esriSelectionResultNew, false);// esriSelectionType.esriSelectionTypeHybrid, esriSelectionOption.esriSelectionOptionNormal, overlayCtx.Workspace); overlaySel = ((IFeatureSelection) overlayLayer).SelectionSet; if (overlaySel.Count == 0) return false; intersectName = intersectCtx.NewFeatureClassName(intersectFeatureClass, overlayFc); gp = new BasicGeoprocessorClass(); intersectFc = gp.Intersect((ITable)sourceLayer, true, (ITable)overlayLayer, true, 0.00002, intersectName); return true; } finally { ComUtil.ReleaseCollection(new object[] { sourceFc,overlayFc,intersectFc, sourceLayer, overlayLayer, intersectName, sourceQf,overlayQf, sourceSel,overlaySel, enumGeometryBind, geometryFactory, sourceGeometries, gp, }); } }

Data and the Monkey Brain

The excellent How the World Works blog by Andrew Leonard over at salon.com links today to a lecture cool enough to make me want to go back to school. (Until I think of the dozens of graduates students who must have given their lives on its behalf...) The lecture is about worldwide economic development data, but its visual presentation is compelling.

At the end of the day, it's just animated scatter plots. But it speaks to the intuitive part of our brain -- and harnesses the massive processing power available in our visual centers -- in a way that tables or static plots never will. I'm always looking for ways to present data in a way that engages the "monkey brain" that doesn't have to think consciously to absorb infromation.

It seems that animation (especially interactive animation) is one of the latest bag of tricks in data presentation. Perhaps this is because modern graphics hardware and internet bandwidth make animation of even large amounts of data quite simple. The next big thing in mapping is animated time series. I don't think it's a coincidence.

The company which makes the world's development data "understandable, enjoyable, and free" -- and the snazzy viewers in the video -- can be found at Gapminder.

Mr. TiVo come here, I need you

The TiVo I got for my birthday has turned on, booted up, and started to play nice with my cable box. (Thank goodness, the cable box supports a serial-port input for channel changing -- no infrared dongles for me.) But even though this new generation of TiVos supports wireless broadband access, it has to be set up with a good old 20th century phone line. Blame the previous owners/remodelers of my house, but there is only one phone line in the place, and it's nowhere near the TiVo. So I'm now watching the TiVo talk to the mothership via about 50 feet of phone line dangling across chairs, sofas, and bookcases clear across the house. Oh well, either I'll get the wireless figured out or train the dogs not to trip on phone lines.