It is what it is

Installing ArcIMS 9.3 on Vista or Vista 64: Internal Error 2738

noreply@blogger.com (Sebastian Good) — Wed, 27 Aug 2008 22:14:00 +0000

A quick note to the web: if you are trying to install ArcIMS 9.3 on Vista, you may run into the cryptic fatal error "Internal Error 2738". Apparently the ESRI installers require VB Script (say it ain't so!) and Vista doesn't register it by default (perhaps a security hole they wanted to plug?). You have to register it yourself by opening an administrator command prompt and running "regsvr32 vbscript.dll".

If you're on Vista, it's probably in C:\Windows\System32. If you're on Vista 64 there's one there but it won't do you any good, as the installer runs as a 32 bit process. (As you recall, System32 holds 64-bit DLLs on Vista 64. Makes perfect sense, right?) You'll need to register the vbscript.dll in C:\Windows\SysWOW64.

Thanks to Swapna at ESRI support services for resolving this issue in just a couple of hours on an otherwise quiet Wednesday afternoon.

Okay, who broke Georgia?

noreply@blogger.com (Sebastian Good) — Thu, 21 Aug 2008 17:04:00 +0000

The US has been "at war" for so long that I suspect most Americans don't think much of the current tussle over Georgia. While the stress of our wars (against drugs, against the Taliban, and against Iraqi militias) have stretched the military to its limits and hurt our economy, the fact is that most people are still not directly affected by them. Why would the threat of a war over Georgia worry us? The number of casualties has not impressed itself into our national consciousness yet. How could it? With approximately 30,000 wounded in a population of 300,000,000 (one hundredth of a percent), the Iraq war cannot affect our conscience the same way the Vietnam war (330,000 casualties in a population of 200,000,000, a fraction 15 times bigger) or World War II (875,000 casualties in a population of 130,000,000, fraction 70 times as large) did? But shouldn't the thought of war with Russia just scare the pants off everyone? Even a limited nuclear exchange would make the numbers above seem like the good old days. Why does no one talk about it?

An increasingly common thread is the one nicely summed up by Thomas Friedman at the NYT wondering why we would "cram NATO expansion down the Russians’ throats". Why was Georgia on a track for NATO membership? Why is Poland in NATO? Against the Soviet Union's stated goals of world domination promising to defend a divided Germany or a weakened United Kingdom with nuclear weapons seemed sensible enough. Against a modern Russia more keen to misspend its oil wealth and struggle with its shrinking population, why do we need to commit American troops to defend Poland, a land with no natural borders? Or Georgia, a charming yet tiny republic of limited geopolitical consequence? (Yes, they have an oil pipeline. There's only so often Russia can threaten to not sell their oil. It's all they've got as a source of national wealth.)

So I've been wondering this for some time. Why do we need to expand NATO to Russia's doorstep? So we look like fools when the question of "will you defend peripheral NATO members with nuclear force" comes up? If Russia invaded some separatists parts of Poland would we really pull a half million troops from Iraq, steam the Navy into the North Sea, and target our nukes at Moscow? Really?. It seems unimaginable.

Reading this article made me remember I've been wondering this a long time. Through the miracles of the Internet and, coincidentally, the Israeli embassy in the US, I found a transcript of me asking this question to then Secretary-of-State Madeline Albright over ten years ago. The full transcript is here.

(My question) It is clear that NATO's role must change in the post-Cold War era, but why does the administration think that expanding NATO's unilateral defense agreements into the former Soviet Bloc is a good idea? Russia's recent pressure on Belarus and some warmongering by its Generals indicate that even an unsteady Russia is not keen on the idea. In this century, Western Europe has offered defense of nations of Eastern Europe before and then reneged on those promises. Why emphasize military inclusion now rather than concentrate on economic inclusion and/or aid?

Her answer, emphasis mine

This is a very important question and it is clearly among the highest priorities that President Clinton has. He has stated that an undivided and stable Europe is very important to the United States.

I think that we all know as students of history that Central and Eastern Europe have, in fact, been the breeding ground of two World Wars. An instability in that region is something that concerns us all. We have made the decision that it is important to expand NATO to cover that region. We, however, also know that it is very important that the Russians do not feel that an expanded NATO is a threat to them or an adversarial move.

The purpose of an expanded NATO is, in fact, to create, or help to create, stability and deal with problems within that gray zone, that gray area in Central and Eastern Europe. We think that that is not only to our advantage but, frankly, also to the advantage of Russia. Because we are concerned about Russia and not letting that great country have a sense that it is being left out, we are also in the process of negotiating a charter between NATO and Russia which would, in fact, have the Russians understand that NATO itself is not an adversary.

How silly does that statement seem now with Russia making nuclear threats against Poland's new US-supplied missile defense system? Back in 1997, oil was about $15/barrel, Russia was divided and confused, and we were pressing our advantage in the cold war. With oil now at $115/barrel, and US power at a nadir thanks to poorly chosen wars at our periphery, how smart does it look to set up nuclear outposts in the bear's back yard? Why we might imagine how we'd feel if the Soviet Union had tried to place nuclear missiles 90 miles from Miami. Fences can indeed make good neighbors, and we ought to be happy to let Georgia be a fence.

Russia won't be strong forever. If we feel threatened by them, let's concentrate on worrying about what gives them strength: the high price of oil. We can wait out their inevitable population decline and foster responsible economic development. If we're going to make a multi-generational geopolitical bet, let's bet on demographics, work on energy independence, and reap the dividends of peace. Let's not bet with our youth that we can push Russia as far as we want, and that all wars are easily fightable on credit. We've already pushed them enough. A war with Russia can't be put on the nation's credit card. I think we knew this 10 years ago when we were uneasy with NATO expansion into the former Warsaw pact. Let's not lose sight of it again.

ArcGIS Explorer Build 480: Threading!

noreply@blogger.com (Sebastian Good) — Wed, 25 Jun 2008 22:02:00 +0000

ArcGIS Explorer (AGX to its friends, apparently) has a new release out, with a great number of promised new features. I'd watched previous builds of AGX through an HTTP proxy to see what all it got up to while fetching images from the web. Investigation showed that it was brutally single-threaded and not terribly bright about what order to retrieve tiles in.

Build 480 changes those things a great deal for the better. Even when looking at a single layer, such as the default satellite/aerial imagery provided, you can see AGX running 2 to 4 simultaneous connections to fetch tiles. Their documentation promises multi-threaded downloads only on a dual core machine, which I've got, so your mileage may vary. I would have thought that even a single core machine would benefit from having several HTTP requests in flight, as they involve so much waiting around, but one suspects ESRI did some performance testing.

When connecting to a very slow service, such as most of the WMS servers out there, I was able to see AGX with as many as 10 connections in flight at once. This is good! There are still some starvation issues (the crappy WMS service kept the fairly snappy ESRI service from showing up for a long time), but this is a great improvement.

The order in which tiles are retrieved still seems suspect to me, with the world view first fetching Asia, the north pole, and Antarctica before the western hemisphere view which is the default. Grabbing multiple images at once compensates for the imperfect tile fetching strategy. And like before, shutting down AGX while it has several open HTTP connections is not pretty: it waits until they are timed out to shut down completely (even continuing to fetch new tiles while it tries). These flaws mean that you still need to be careful using any service which is slow or broken -- other imagery will get stuck behind it.

More news as investigations continue. So far, so encouraging!

Drilling for new oil in the USA

noreply@blogger.com (Sebastian Good) — Wed, 18 Jun 2008 15:13:00 +0000

So the debate over drilling within the US has opened up again. There will be lots of debate about environmental impact that I don't feel sufficiently informed on to comment about. But given the success Bush has had redefining US foreign policy as a contest to see who can be the biggest asshole (read: tough on terrorists), I suspect the debate will revolve once again around that old canard: "energy independence". The idea is that if we drill more at home, we'll be less dependent on foreign oil.

Let me present a slightly different viewpoint I don't hear discussed much in the media. Allow me a couple of asides:

First, the notion that oil drilled in the United States will be consumed in the United States is of course not entirely true. Oil produced in Alaska may be much more cheaply consumed in Japan than in New York. That's just geometry. The market will make that decision based on shipping distances, quality of crude, availability of refining, and a thousand other factors. What domestic drilling may do is increase the world's supply of oil a bit (or at least slightly arrest the decline of US production), and thereby place downward pressure on the price of oil. This seems like a laudable goal.

Second, you will hear a few people say that it's not worth drilling for more oil because the amount is tiny and would not be felt for many years. Just because something won't have an effect for 5 years doesn't mean it shouldn't be done. But it's not often appreciated how much oil we consume and how much impact US exploration might have. Of course the numbers are always open to debate, but take the two extremely massive oil fields recently discovered off the coast of Brazil. These are game-changing fields, some of the largest on the planet. They total roughly 10-20 billion barrels of oil equivalent. That's a lot, right? What if we found those hiding in the Gulf of Mexico. Even though we're pretty certain there's nothing that big out there, even so... These fields, if drained completely dry, would only serve the US's oil consumption for less than three years. Three years. The numbers are staggering. Drilling activists like to suggest that there is a ton of oil in the ground that the tree-huggers are just hiding from us. That's true: but the amount of oil is not particularly significant if you look at it from a multi-decade point of view.

Which brings me to the point I was really trying to make. The point is actually very simple.

There's not a lot of oil left at reasonable prices given current consumption and growth trends. It's very hard to tell whether this price spike is the beginning of the end, or just a head fake, but the end is coming in our lifetimes. As any petroleum geologist will tell you, "the end" will arrive with lots of oil left in the ground, just too expensive for our means. When that end comes, when oil is $1000 per barrel, gasoline is rationed for national security reasons, when poorer countries without access to alternative energy technology are going to war to secure the oil they need to fertilize their crops so they can eat and drink, what situation do you want the US to be in? With major reserves already tapped to secure a few extra years of $4/gallon gasoline? Or with major reserves available within our borders to provide the fuel the army and navy require to secure peace in this dangerous world? Do you want the strategic oil reserve to have been run down to keep the cost of Summer road trips low? Or in place to ensure the smooth functioning of the military when Mexico, Venezuela, and Nigeria won't have any oil left to export? Do you want to have burned all the oil to light up Starbucks signs at night, or have some left over to maintain the crop yields which allow our nation to produce something the rest of the world thinks is worth buying?

Oil is running out. Aside from the obvious implication that we should be working on alternative sources of energy (and oil at $140/bbl is doing that much better than any Congressional plan would), it's not obvious to very many people that we should keep what's left for ourselves. Every nation should be looking towards energy security, just as they look for food security. Why should we pawn our future for a few years of fun in the present?

Greenspun Tenth Rule Redux: Vista is Terrible

noreply@blogger.com (Sebastian Good) — Sat, 14 Jun 2008 04:53:00 +0000

I just got a new laptop with Vista 64 on it. In lieu of flowers, please send donations to the ACLU. But I digress.

It turns out you can't install Visual Studio 2005 from a mount point in Vista. You know, I thought it would be convenient to move on from the bad-idea-when-it-started-14-years-ago of always agreeing to map the same drive letter on each of my computers to the same actual share I maintain of software, music, etc. I thought, hey, Vista supports symbolic links. I'll map c:/users/public/software to \\myserver\public\software. And I'll install from there. No drive mappings to remember to add, remove, wait for Explorer to hang from. I'm afraid not. The VS 2005 install just fails. Why? How can it even possibly tell the difference?

Here's the kicker. Even when I mapped my F: drive like we used to do in the gay nineties, the install failed. The amazing part? The install log still complained about not being able to read from c:/users/public/software. Deep down in the system, it knew that F: was also mounted to that point on my C: drive! How could it be? Neither F: nor c:\users\public\software are the 'real' location of that share. Good lord. How does Microsoft wait 20 years to implement links and then get them wrong?

It reminds me of Greenspun's tenth rule of programming:

Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified bug-ridden slow implementation of half of Common Lisp.

To which I should add: Any sufficiently mature operating system contains an ad-hoc informally-specified bug-ridden implementation of half of Unix. And in the great spirit of Raymond Chen, pre-emptive snarky comment: yes, that probably applies to most Unix implementations.

SDE 9.2's ST_GEOMETRY: Part Two, The Empire Strikes Back

noreply@blogger.com (Sebastian Good) — Mon, 05 May 2008 22:05:00 +0000

I've been investigating ESRI SDE's ST_GEOMETRY support and performance on some simple sample data sets. I'm embarassed to say I don't recall where they came from, but they are all US counties (about 3,000) and all US ZIP codes (about 30,000). My initial reactions were excitement at the speed of spatial joins on small datasets, disappointment at slow performance on large datasets, followed by theorizing about the cost of out-of-process calls.

Well, in these things it's not the journey, it's the destination. I thought I better compare to Oracle Spatial's SDO_GEOMETRY to make sure I was comparing apples to apples, as it were. The results are interesting.

I'm trying two simple tests of throughput, one a spatial join which selects all the ZIP codes which overlap a given (semi-random) set of counties, the other a raw select which forces all geometries to be converted to WKB format for consumption by a putative 3rd party tool.

For Oracle


select count(*)
 from ozip zi
 join ocounty co on (sdo_relate(zi.shape, co.shape, 'mask=ANYINTERACT') = 'TRUE')
where co.objectid between 1200 and 1500;
select sum(dbms_lob.getlength(sdo_util.to_wkbgeometry(shape))) from ozip;

For SDE


select count(*)
 from zip zi
 join county co on (st_intersects(zi.shape, co.shape)=1)
where co.objectid between 1200 and 1500;
select sum(dbms_lob.getlength(sde.st_asbinary(shape))) from zip;

What should the baseline be? I figure it's the existing geoprocessing and rendering tools from ESRI. So for the spatial join I just used the Intersect toolbox on the feature classes with the 'objectid between 1200 and 1500' as the definition query for counties, as above. For raw rendering, I simply averaged 3 successive refreshes of a map full of all 30,000 zip codes. This makes the baseline look even slower than it really is, as it also times rendering and context switches. SDE's fast, people; we knew that.

The somewhat surprising results are as follows (all times in seconds).

Operation	SDE/SDO	SDE/ST	ST_GEOMETRY	SDO_GEOMETRY
Join	25	25	6.0	?
Scan	15	5	35	∞

I got tired of waiting after 15 minutes for Oracle to convert its 30,000 zip codes to WKB, so I gave it a score of ∞. All queries appeared to be CPU bound, which makes sense as the entire data set fits into the memory of even this old laptop.

Update 2008-May-5: In amusing and ironic twist, Paul Ramsey has notified me that Oracle doesn't let you release benchmark results without making sure they've been dolled up and faked by their sales engineers, er, excuse me, cleared by their legal department. I've removed some of the actual figures from the charts above. Infinity's hard to hide in a closet, even with slick marketing.

The lessons are certainly mixed.

Joins. For the kinds of in-SQL-on-the-fly joins that make geometry data types tempting, SDO_GEOMETRY might be a clear winner (better ask Oracle legal) but not by a massive amount (?? seconds versus 6.0 seconds, making ST_GEOMETRY ??% slower). Perhaps this is because of the process switches endured by implementing ST_GEOMETRY in st_shapelib, perhaps not. Perhaps it is the difference in indexing schemes. More complex and varied tests would need to be done. For back-of-the-envelope estimates, they're roughly equivalent. For most applications, if 10 seconds is acceptable, so is ??. ST_GEOMETRY and SDO_GEOMETRY are both certainly far smarter than the Intersect toolbox, which spends most of its time querying more data than it needs.

Scans. If you want to grab data from these systems and process them using open standards, you lose either way. With ST_GEOMETRY, you can do the work using ST_ASBINARY, but the performance is unimpressive: roughly 1000 shapes per second. (Compare at 6000 per second for SDE querying ST_GEOMETRY itself and ArcMap rendering them.) SDO_GEOMETRY goes completely to lunch; they're not taking WKB very seriously. My 3:1 showing for SDE rendering of ST_GEOMETRY versus SDO_GEOMETRY layers throws a lot of these numbers into question. Would ESRI really let their Oracle Spatial implementation be that much slower?

My provisional conclusion is that ST_GEOMETRY holds promise for spatial SQL, as much as SDO_GEOMETRY, though it probably needs more tuning from ESRI's side. No one is going to be writing useful GIS tools which use the WKB/WKT forms of these geometries anytime soon. If you want fast scanning of data, you've got to get under the covers and read the data natively. That is, I suppose, the next experiment. I'd like to know how easy it will be to read ST_GEOMETRY data natively to .NET, where my particular bread is buttered. More to come.

SDE 9.2's ST_GEOMETRY: Part One, perhaps Part Last

noreply@blogger.com (Sebastian Good) — Mon, 05 May 2008 02:44:00 +0000

Last time, I played a little with SDE 9.2's new ST_GEOMETRY support and was glad I waited for SP5 (!). More experimentation revealed that writing spatial SQL was easy. I've waited a long time to use ESRI-sanctioned methods for asking things like


select *
  from zip zi
  join county co on (st_intersects(zi.shape, co.shape)=1)
 where co.population > 1000000;

And it does work. The spatial indexing gets used intelligently, and you can throw (variants of the above) into ArcMap and render spatial queries on the fly. I even started experimenting with doing on-the-fly geoprocessing with things like shape intersections.

I soon noticed that doing anything with more than a few hundred shapes was slow. Of course at first I blamed my queries, the Oracle optimizer, the spatial indexes, anything. But this ain't my first rodeo. That stuff was all fine. No, I was noticing that no matter what I did, ST_GEOMETRY couldn't deal with more than about 1000 shapes per second. Fancy spatial indexes don't do much good when the final geometry-to-geometry filter maxes out on so few shapes.

Why oh why?

Part of the problem seems to be because all ST_GEOMETRY functionality is implemented with an external C DLL, st_shapelib. This is how Oracle wants you to do it. Every call to an ST_GEOMETRY function is made out of process from the Oracle process serving your connection (oracle.exe on Windows) to a spawned executable (extproc.exe on Windows) via pipes or sockets. When intercepting a CREATE TABLE command and jumping in to create an index, these two process switches are no big deal. When determining whether one polygon overlaps another during a join operation over tens of thousands of individual polygons, it is murder. Quantifying the cost of a context switch is tricky, but at the very least it is thousands of cycles. Figure four thousand cycles, plus a few thousand for copying the data back (and forth), plus the (perhaps greater) cost of killing cache locality and dumping all your registers, and it's not a pretty picture. Compare that to the cost of doing the few dozen or hundred operations needed for determining overlap of my typical polygons and it's likely this out-of-process trick is killing performance by a factor of ten or more.

To make sure this wasn't just a fancy theory, I pulled up trusty perfmon.exe and had it count context switches. Here it is ticking along for a while on my idle machine, then executing a very simple query for a couple of seconds.

Can you spot the query? The average number of context switches per second hovers around 850, then spikes to 20,000 during the query, then back.

For completeness, I wrote a test program which simply created two threads which did nothing but relinquish control and loop, trying to get an upper bound on the number of switches I could do. Trusty F# with its concision was useful here.


open System.Threading;;
let rec cswitch () = Thread.Sleep(0); cswitch();;
let makethread () = (new Thread(cswitch)).Start();;
makethread();;
makethread();;

And I was off to about a million context switches per second. That's an unrealistically high number, since the above code is just an infinite loop with no overhead whatsoever. But I would guess that 1 million no-overhead process switches translates to roughly the ~100,000 order of magnitude real-world process switches I was seeing with the st_shapelib calls. I also suspect it means st_shapelib is doing more than a trivial amount of work, which might mean there's room for performance enhancements, too. Just don't know.

But the context switching seems to be a fatal flaw. Oracle's EXTPROC isn't meant for this kind of fine-grained work. The next investigations involve checking out whether Oracle can be sweet talked into running these external processes in-process, but I'm willing to guess the answer is no. So, it was promising, but there's no way this architectural decision gives anyone the kind of speed they're looking for.

It appears you can't write code in C in Oracle without being sent off to EXTPROC, so perhaps Oracle Spatial will suffer from the same problems. That'll be the next set of investigations.

(This post was updated a few hours after being published with a few more thoughts on Oracle extensions.)

SDE 9.2's ST_GEOMETRY: Part Zero

noreply@blogger.com (Sebastian Good) — Fri, 02 May 2008 22:30:00 +0000

It was over a year ago I sipped the ESRI ST_GEOMETRY Kool-Aid. Fiteen months later, things finally worked out where I could try out some small examples. This should be the first in a series of posts. We'll see. In brief, I'm glad I waited a year to try, and more time yet may be in order, depending on what you're trying to do.

Installing SDE 9.2 is easy enough, and creating tables which use the ST_GEOMETRY type is easy, too. You just, you know, do it.

CREATE TABLE THINGS (X INT, SHAPE ST_GEOMETRY)

The critical documentation is called Working with a Geodatabase using SQL. It's easy to find the basic operations.

What's not as easy is a few administrative details. First, setting up spatial references is no picnic. If you load data via ArcGIS, it magically gets taken care of for you. If you're loading it manually, you need to create your own spatial references, including resolution, offsets, and the whole nine yards. I haven't done that yet, and have been happy to let utilities create them. Unfortunately, these utilities create a custom SRID for each dataset I load, even though they have the same coordinate system. Each SRID has its own offsets and grid spacings. I'll have to figure out how to force data to be loaded into existing SRIDs, probably by labeling the column in SDE's metadata. I loaded data using shp2sde, which is certainly a cop out.

Then comes querying data. If you just run and try "select st_astext(shape) from zip", you may get an interesting listener error: ORA-28575: unable to open RPC connection to external procedure agent. It turns out all the ESRI ST_GEOMETRY functions are implemented using an out-of-process procedure call, which the Oracle listener has to be configured to support. It comes this way with normal installs, but somewhere along the line I'd lost mine. Never mind, ESRI has a good article explaining what to do.

Now comes the part where I'm glad I waited a year. Doing any of the most trivial operations, such as the aforementioned "select st_astext(shape)..." resulted in a host of clear nastiness, the most common of which was ORA-28579: network error during callback from external procedure agent. A lot of hunting finally revealed that these bugs were only fixed in SDE SP5. (My money says they weren't all fixed; we'll see.) It turns out SP5 for Oracle 10g R2 was withdrawn a few days ago because of a nasty regression. Now this regression was unrelated to ST_GEOMETRY support and Friday afternoons of playing around with technology don't come around very often, so I was very relieved to find that James Fee had a copy of it still available. That fixed the bugs so I could start doing some real testing.

More details in the next post, but first impressions are that doing some casual spatial joins (e.g. setting a where clause in ArcMap to only show cities which intersect Harris county) seems pleasantly fast. (Update, 2008-May-5: Further investigations reveal this was only fast on small datasets. Spatial joins are slow, too.) Selecting the data as WKB is intolerably and comically slow. This seems to be because the call to functions like st_asbinary is made out of process to Oracle's extproc.exe. I am currently testing on an old single core laptop, so it may be that a dual processor machine would handle more process switches per second, but it probably goes without saying that calling an out-of-process function to convert a few kilobytes of binary data from one form to another is extravagant. If this is the case, then while the spatial indexing and intersection type functions will be immensely useful, there appears to be no future (on Oracle, at least) at all in writing tools which actually go against the WKB representation made possible by SDE.ST_GEOMETRY. Unless you're happy to query less than 1000 rows per second.

SDE, of course, goes "under the covers" when rendering data from ST_GEOMETRY columns, and so its plenty fast. Something to ponder.

Until next time.

C2860 and C++/CLI

noreply@blogger.com (Sebastian Good) — Thu, 24 Apr 2008 18:07:00 +0000

Just for future Googlers, it is possible to get a compile error from the C++/CLI compiler (that is, Visual Studio 2005 compiling C++ code with the /clr switch) which looks like

C2860: 'void' cannot be an argument type, except for '(void)'

This happens at absolutely random places where you reference .NET objects that it should already know about. (The error will further read, if you look at the detailed compiler output, something like " This diagnostic occurred while importing type 'Fred::Ethel ' from assembly 'cppclitestingfs, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'.")

Muh? It turns out in this case the class I was using used a class in its definition a class which came from another assembly (FSharp.Core, as it turns out, but it could have been anyone.) Instead of saying something useful like 'Fred::Ethel references type Option<A> which is not defined', it said ''void' cannot be an argument type'. Obviously.

My completely un-called for guess? Someone on the C++/CLI compiler team substituted the word 'void', perhaps by accident, when an imported type had an unresolved external reference. Another compiler pass downstream complained quite rightly about this, but lacked context to give a more meaningful message. Is there a place one submits bugs in Microsoft compilers that is not informationally equivalent to a black hole?

That was Easy! Canceling a credit card

noreply@blogger.com (Sebastian Good) — Mon, 31 Mar 2008 21:41:00 +0000

It must be a sign that I don't spend enough money. I called American Express today to cancel a credit card. I hadn't used it in a few years, so I didn't expect them to fight much. Imagine my surprise when "cancel a card" was a second-level item on their automated telephony system! Most companies make it impossible to find their phone number, never mind make it easy to find out how to cancel your relationship with them. I was further flabbergasted when I found out that my cancellation merely required me pressing a number on my phone; no high pressure tactics from call center salesmen! It almost made me feel bad to have canceled when the company provided such good customer service! But never mind, it makes me want to keep the existing Amex charge card I have even more: it's too damn hard to find good customer service these days.

Different References between Debug and Release Builds in Visual Studio 2005

noreply@blogger.com (Sebastian Good) — Thu, 27 Mar 2008 22:19:00 +0000

What if you want to reference different DLLs in Debug and Release builds of your favorite .NET project? Perhaps you've got some unmanaged DLL spat out by a system that you don't control, but that produces different builds? Or, as in my case, you've got an F# assembly you want to reference from a C# assembly, but F# doesn't yet support "project references", so you have to link directly to the DLL. I'm not convinced what I did will always work, but it seems to right now.

The references in your project (in this case a C# project) are held in a tag called <ItemGroup>. It turns out you can add restrictions to that tag just like you can the other tags in your project, e.g. giving your assembly different names in debug and release modes. So you can end up with something like this.


  <ItemGroup>
    <Reference Include="System" />
    <Reference Include="System.configuration" />
  </ItemGroup>
  <ItemGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
    <Reference Include="my favorite debug dll">
      <SpecificVersion>False</SpecificVersion>
      <HintPath>..\..\Debug\so-and-so.dll</HintPath>
    </Reference>
  </ItemGroup>
  <ItemGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
  <Reference Include="my favorite debug dll">
      <SpecificVersion>False</SpecificVersion>
      <HintPath>..\..\Release\so-and-so.dll</HintPath>
    </Reference>
  </ItemGroup>

The smoothly oiled Starbucks PR Machine

noreply@blogger.com (Sebastian Good) — Tue, 04 Mar 2008 05:31:00 +0000

Starbucks closed recently to retrain their baristas. Why was this necessary? A massive overexpansion, poor training program, sluggish growth, focus on sandwiches and music instead of coffe, and competition from Dunkin Donuts and McDonalds? It seems their US president was shown the door today. Not pretty.

But the PR machine is operating perfectly. The cover story was that Starbucks was going to "revive the intimate, friendly feel of a neighborhood coffee shop". You mean the ones they try to put out of business? Still, the entire planet was notified of the planned three hour closing. It was considered national news in the United States, complete with radio and TV interviews of Starbucks staff. Tens of millions of dollars of free advertising in return for shutting down during three slow late evening hours. I thought that was smart.

But what if you were thinking of working at Starbucks and read about the training? Apparently it was pretty medieval. Who wants to take a job where you have to be trained to be snob? Seriously, and at a part-time fast food job? Forget it.

No problem. Starbucks staff get health benefits, and no one should forget that. What if there was a health-related story about just how amazingly cool Starbucks staff were? What if the story were so irresistibly sweet that it would have to make the national papers, just like the shut down? This might heal any potential damage the shut down did, right? That would be smart!

Shoot on over to today's human interest piece in the New York Times about, get this, a Starbucks barista who donated her kidney to a customer. Not to take away from the impressive generosity this woman showed, but why didn't this story make the rounds last fall when it happened? Right in time for the Christmas season perhaps?

Nah, probably better that it showed up today to either remind workers that Starbucks is great, even though they train the hell out of you, or to totally bury the news that their US president got the sack today.

Say what you like about their coffee, their business practices, or their organ donors, but their PR department kicks all ass. It almost makes you feel sorry for the schmucks at the New York Times who swallowed this hook, line and sinker instead of reporting on, oh, the massive cost of the war or something useful. Almost.

Postscript: There is apparently some evidence that Starbucks's ubiquity leads to an expansion of demand for coffee so large that many neighborhood coffee shops have increased sales after a Starbucks moves in across the street... Anyone wonder who planted that story?

Randy Newman Rocks

noreply@blogger.com (Sebastian Good) — Tue, 04 Mar 2008 02:47:00 +0000

Randy Newman is brilliant, but has anyone else noticed that the intro to his main Monsters Inc. theme is the same as the intro to the famous ragtime hit Temptation Rag? The arrangement is even identical to the Benny Goodman version, with clarinet and xylophone melody.

How about that?

Huh. Mono Just Works.

noreply@blogger.com (Sebastian Good) — Thu, 28 Feb 2008 04:05:00 +0000

So I continue work on a project which I want to write in .NET but needs to run on Red Hat Enterprise Linux (an ancient version, I'm sure) and Vista. And my home computer is a Mac. Surely this is a recipe for Mono.

I started gingerly, programming in Visual Studio 2005 and building comprehensive unit tests. I run them with Test Runner or NUnit directly in our automated build. Fine.

Then I got a little braver. I installed Mono on our Linux machines. (Long story, actually, and not very amusing. It had to be compiled from source and one of the source files edited because this version of Linux is so old some of the thread functions now have different signatures. But never mind.) Not wanting to jump in whole hog, I simply copied the compiled DLLs from the Windows machine to the Linux machine. Mono comes with a version of NUnit, so I started it up.


linux-machine:/glb/home/myuserid: bin/nunit-console2 MyProject.dll
NUnit version 2.2.0
Copyright (C) 2002-2003 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov, Charlie Poole.
Copyright (C) 2000-2003 Philip Craig.
All Rights Reserved.

OS Version: Unix 2.4.21.47    Mono Version: 2.0.50727.42

......

Tests run: 6 (all pass), Not run: 0, Time: 3.859981 seconds

Tests run: 6, Failures: 0, Not run: 0, Time: 3.859981 seconds

Wow. Just like that. This code isn't exactly running a nuclear reactor, but it is out there multithreading, doing file I/O (with locks), XML serialization, and all manner of fun. Mono even gets Linux's different file naming conventions correct, and thanks to defensive coding of file-handling functions, no change needed on my part. (Hint: learn to love System.IO.Path).

Well, if it works on Very Old Red Hat Linux, surely it'll just work on Mac OS X, right? Yes, why not. I did "svn up" and got the code and binaries down to my little Mac called, er, LittleMac. nunit-console2 reported the same happy results.

One binary, three runtimes.

But why stop there? Why not try the Mono compilers, too? Other than masochism, no particular reason, but I was intrigued by the idea of being able to debug in Linux. After all, the programmers this is supplied to will probably want to occasionally debug the code, and they'll want to do it in Linux as well as Windows. Why not try MonoDevelop? Heck, it's included with Mono downloads. Turns out it's file format compatible with the .sln and .csproj files Visual Studio 2005 uses. I just opened it on the source code I'd downloaded from Windows. I pressed Build, then ran nunit-console2 again. Same results. That was on the Mac.

It just worked. I'm sure my honeymoon will be over soon as I'm just scratching the surface, but I'm very encouraged. Will the auto-generated C++ wrappers for my C# code work just as pleasantly on all these platforms? Stay tuned.

Skynet Begins Self Construction

noreply@blogger.com (Sebastian Good) — Tue, 26 Feb 2008 04:01:00 +0000

Yep, Google is now constrained by the very infrastructure of the Internet. From the Official Google Blog.

... One of the biggest challenges we face is staying ahead of our broadband capacity needs, especially across Asia ... Collectively we just signed an agreement to build a new high-bandwidth subsea cable system linking the U.S. and Japan.

I think there's a couple of interesting points here.

Google's demand on bandwidth is greater than any one company can provide. This seems to give it a great deal of leverage, as it probably makes a higher margin converting that bandwidth to advertising than the telcos make converting boats and electricity into bandwidth. Google's incessant preaching about how it just wants an open Internet to compete on is probably true. The bigger the Internet gets, the more money they make. But clearly they have switched from roughly passive beneficiaries of Internet advancement to aggressively self-interested builders of the Internet.
While Google may not be getting into the telco business (they emphatically deny it in this release) their leverage and reach make them natural competitors to the telephone companies. Everyone would love to see the telcos suffer for their crappy service and monopoly behavior. But the telcos were granted their monopoly by the government and so still have to (theoretically) abide by serious regulations. Google earned their monopoly all by themeselves. Who are they beholden to?
The speed of light is still a hard limit. Why isn't Google building more data centers in Japan? Why a big cable from the US to Japan? Is there that much youtube content generated in one country and then viewed in another? Perhaps, but it makes me wonder whether Google's internal traffic needs are on a par with or greater than its external traffic needs. That's pure speculation.
Where I came up, a "couple" can mean more than two things.

When will Skynet become sentient? I console myself that any soul who comes into being through reading the Internet will probably be so confused as to be harmless. Will it be addicted to porn? Unable to rectify the political rants from right to left? Get into Wikipedia editing contests with itself? Let's just hope the launch codes for the nukes aren't online anywhere.

ArcMap Automation: Man the Message Pumps!

noreply@blogger.com (Sebastian Good) — Fri, 22 Feb 2008 00:10:00 +0000

The PUG is coming up, so it seems an opportune time to serve a little amuse bouche to get people in the mood. We write a desktop application which wants very much to be an ArcMap plugin. It creates layers, maps, and generally talks to people's real live ArcMap sessions. But we don't want to be an ArcMap plugin, because frankly the world's a lot bigger than ArcMap. We do other stuff, too. So? We automate ArcMap.

We tickle it remotely. You want to create IFeatureLayers in a map? You can do it cross-process, thanks to the beauty of DCOM. We do this in C# where it doesn't blow our minds, and it works. It's actually pretty clever.

ESRI's main applications (like ArcMap) expose an IObjectFactory interface, through which you can create remote objects. We abstract that so that our functions can run either on our server (creating local map documents on the fly) or on a desktop (tickling ArcMap remotely. Like so.


public static object Create(Type type, IObjectFactory factory)
{
 string progId = GetProgId(type);
 return factory == null
  ? Activator.CreateInstance(Type.GetTypeFromProgID(progId))
  : factory.Create(progId);   
}

Cool. So you can use this factory to create layers remotely; it looks like the code you'd write in a plugin, but you don't have to be a plugin. For instance


public static IGeoFeatureLayer CreateFeatureLayer(
  bool visible, 
  string layerName, 
  string layerDescription,
  IFeatureClass featureClass,
  IFeatureRenderer renderer,
  bool legendVisible,
  string legendLabel, 
  string labelExpression,
  bool isLabelled, 
  string displayField, 
  bool showTips,
  IObjectFactory factory)
{
 IGeoFeatureLayer layer = (IGeoFeatureLayer)InteropUtil.Create(typeof(FeatureLayer), factory);

 layer.Visible = visible;
 layer.Name = layerName;
 if (layerDescription != null) ((ILayerGeneralProperties)layer).LayerDescription = layerDescription;
 layer.Renderer = renderer;
 SetLayerLegendProperties(layer, legendVisible, legendLabel);
 SetFeatureLayerLabel(layer, labelExpression, isLabelled, factory);
 if (displayField != null) layer.DisplayField = displayField;
 layer.ShowTips = showTips;
 layer.FeatureClass = featureClass;
 return layer;
}

Brilliant. You're in, but you're not a prisoner. The exact same approach can be used for automating MS Office. It's built this way on purpose, and it's cool. Well there's a twist. You can write this code, and you can call it, but it usually doesn't run. It takes minutes or hours to complete, if it completes at all.

You scratch your head. You ponder. Then, absolutely randomly, you notice that when you move your mouse repeatedly over the ArcMap window, it finishes faster. Surely this can't be, you say. I'm being fooled by randomness. But this persists.

Finally you break out the DCOM scriptures and note that DCOM messages between processes arrive as windows messages, just like Alt-Tab requests, minimize requests, and yes, mouse movements! Could the DCOM messages your app is sending be stalled upon arrival to ArcMap?

It turns out the answer is yes. ArcMap does not properly receive these DCOM messages without intervention. You have to force them to pump their Windows message pump. Moving the mouse does that; it's just a nice benefit that while responding to the mouse move messages, they also process the DCOM calls you're making to create layers.

Telling users to scribble over the ArcMap process with their mouse while we were talking to it seemed like it'd get a lot of laughs in our training courses, and not the good kind. So what else could we do? We wrote a function which, while the DCOM call is pending, pelts ArcMap's message pump with messages, forcing it to work. It's like screaming at the top of your lungs "PAY ATTENTION PAY ATTENTION PAY ATTENTION TO ME!!!!". Hey, who are we to argue with a prima donna?

Thusly:


/// example: using(new RemoteArcObjectsPump(app.HWND) { ... }
public class RemoteArcObjectsPump : IDisposable
{
 /// Constructs and automatically begins pumping. If HWND is zero, no timer is created.
 public RemoteArcObjectsPump(int hWnd)
 {
  if (hWnd == 0) return;
  _hWnd = hWnd;
  _thread = new Thread(new ThreadStart(Ping));
  _thread.ApartmentState = ApartmentState.MTA;
  _thread.Name = "Remote ArcObjects Pump";
  _thread.IsBackground = true;
  _thread.Start();
 }

 /// Continuously pings target window with null messages to force message pump to process.
 /// By waiting for the message to return (i.e. SendMessage instead of PostMessage) we don't
 /// overwhelm the target with messags to process. However we do want to send another message
 /// just as soon as we hear the old one is processed, hence no sleeping or timers here.
 private void Ping()
 {
  while(true)
  {
   try { Win32.SendMessage(new IntPtr(_hWnd), Win32.WM_NULL, 1, 0, true); }
   catch {}
  }
 }

 #region Dispose
 void IDisposable.Dispose() { Dispose(true); }
 ~RemoteArcObjectsPump() { Dispose(false); }
 private void Dispose( bool disposing )
 {
  if( disposing && _thread != null )
  {
   _thread.Abort();
   _thread = null;
  }
 }
 #endregion

 private Thread _thread;
 private int _hWnd;
}

And you call it in a using. When the object is disposed, it stops pelting the remote ArcMap session with messages. (Oh, you have to wrap the native Win32 SendMessage function to call it from .NET, but that's a nice exercise for the reader.) Upon reflection, it might be more efficient to give this task to the ThreadPool, or keep a dedicated thread around, but this is already extremely heavy lifting. Process switches galore, never mind ArcMap's own general pokiness meant we didn't stress too much about a few threads here and there.

Well, it wasn't cranberries in lobster foam, but we found it amusing. I hope you did too. Merry PUG, and see you there next week!

Wherefore are thou Topology? (A Plague On Both Your Houses!)

noreply@blogger.com (Sebastian Good) — Fri, 08 Feb 2008 16:19:00 +0000

Not a lot of ESRI posts here recently for a mostly happy reason: there's been little ESRI programming in my universe the past few months. I've been working on a fun little wrapper generator for managed code called GIWS (no, not about the Chosen Tribe, it's SWIG backwards. Get it?) Anyway, that's a post for another time.

Today I come not to praise ESRI Topologies but to bury them. One of our products has a feature where you download a little personal geodatabase from our web application to your desktop to do detailed editing in ArcMap, then you send it back to us and we unpack it. We're thinking of scrapping the whole approach as it turns out to make sense to programmers, but not to end users where it is deployed. But that's yet another post for another time.

Anyway, we take care to create the feature classes in this personal geodatabase in a topology, complete with rules about overlaps, to help users create clean datasets. Several of our feature classes are essentially coverages: the polygons need to be non-overlapping. We thought that by creating a topology automatically we'd be doing our users a favor. They're tedious to put together, and somewhat intricate.

Well, two years later we're getting rid of the topologies. I thought it might be worth sharing our reasoning, as I'm curious whether anyone else has found similar problems. (FWIW, we're using 9.0. Doing work for a big company means you have the awesome upside of knowing your work matters on a large scale. The downside is usually being 12 months behind the technology curve. That's okay.)

Topology algorithms find uncorrectable problems. We had many instances of topology scans finding, for example, polygon overlaps, which turned out to be degenerate lines, points, or even invisible artifacts. These seemed to be associated with datasets where original work had been done in a projected coordinate system, then projected into the WGS1984 we use internally. I understand that projection might cause points to snap differently, creating errors. We all get it. But the problems detected would turn out to be invisible or uncorrectable. That aggravated people.
Topology algorithms are different than geoprocessor algorithms. The feature classes users edit were inputs to a series of geoprocessing algorithms. Nothing exciting, mostly intersections followed by some algebra on the attributes. But quite often a topology check would claim no overlaps, while an intersect would show overlaps. (It forced us to do a sanity check before geoprocessing of doing a self-intersect on each layer and asserting that the number of input polygons equalled the number of output polygons.) We did not spend time to figure out who was right -- and given well-known robustness issues in spatial algorithms, it may well be that both are correct. But since our results are created by the geoprocessor, we decided to use it.
Topological Editing is a very advanced skill—people don't like learning it. We had trouble convincing our users to learn the topological editing tools. Heck, normal editing in ArcMap is hard enough. I couldn't really blame them.
Topologies add awesome bloat to geodatabases. We were seeing geodatabases with a dozen simple feature classes bloating to 700MB after editing. Compacting them would take them back to 2MB. Ouch. We know databases need to be compacted now and again, but this was a little much for us.
Toplogies would cause obscure COM errors in our geoprocessor. This one may be sort of our fault. We are using the 9.0 geoprocessor in-process on the main STA thread of our desktop application. It doesn't seem to like that, and we're contemplating running it as an out-of-process python script on demand. Nonetheless, the stability of our tool has increased since we didn't include topologies. Given the above reasons not to use topologies, it wasn't worth debugging this one.

We really wanted topologies to work. They make sense, they reflect how people really think, they ought to be the bee's knees. But we were ultimately disappointed. Perhaps they're better in 9.2 or 9.3, but I doubt we'll try them again. We coded our own overlap detection and repair tool for people who can't use the ones already out there (e.g ET Geowizards).

Hug a Neogeographer Today

noreply@blogger.com (Sebastian Good) — Fri, 14 Dec 2007 04:49:00 +0000

Having had the delightful opportunity to recently hear "neogeographer" bandied around (in person!) like a swear word, I've been keen on the recent discussion involving it in the GIS press, e.g. on the All Points Blog or The Memory Leak.

It reminded me of how much trouble I got into for being such a sourpuss in my analysis of the 2007 ESRI UC keynote session and ESRI's strategy in general. Six months ago, I said

The attitude at the moment seems to be, "If you're using Google or Microsoft for anything, you're clearly an amateur, and you can go $*#& yourself." Of course Grandpa Dangermond would never say that — he let his attack dogs do it on stage, but the effect is the same. True "professionals" will make "authoritative" data and "publish" it to these "free, consumer" services.

I'm not saying I said it first. But that schism is opening wider and wider.

And I'll say it again: GIS departments need to go the way of the VARCHAR department. There will always be an important place for the surveyor and geodesist who really can tell me if I'm sinking a well within 10 meters of the exact seismic trace I had in mind. But the cult of the GIS dinosaurs simply needs to get out of the way of all the millions of people whose creativity and knowledge can be unleashed by "neo"geography tools. The VARCHAR department will still have well-paid experts who can beautifully typeset new hardcovers and textbooks, but it's just not necessary to use desktop publishing software when a TEXTAREA or Twitter post will do.

Hey GIS programmers: barbarians are at the gates! Do us a favor and let a few in!

Mono's VMWare Image: Sometimes a ZIP is not a ZIP

noreply@blogger.com (Sebastian Good) — Fri, 17 Aug 2007 18:06:00 +0000

So it finally came down to it: I wanted to run Linux in order to test some Mono interoperability code. We are writing libraries that will be used in .NET/Windows environments as well as C++/Linux environments, and getting the pointer buggery correct is important. We've got a C++ program (with a 25 year pedigree) that needs to start using components we'd just as soon write using .NET.

This is a case where I am not comfortable just testing some things in Mono on OS X because the issue is not managed code, but specifically unmanaged code. OS X has enough of its own skeletons; I don't want to sweat the wrong details. So I've gone off to the Mono site and downloaded their most recent VMWare image. It downloads in 20 minutes or so and unzips in another five. In a strange twist, I found that if I unzipped it using OS X's command line unzip command, the command complained about corrupted zip entries. VMWare started the unzipped mess properly, but nothing quite worked right correctly in the resultant image. I used the GUI unzip (whatever happens when you double-click the ZIP file in Finder) and it worked like a charm. Humph.

So I guess my Linux world for the next few weeks or months will be openSUSE 10.2, Novell's pet distribution. We'll see how it goes. I love that every few years as I play with Linux I have to learn a new package management library.

What Can a RESTaurant Teach You About REST? (Whataburger)

noreply@blogger.com (Sebastian Good) — Thu, 16 Aug 2007 03:20:00 +0000

We're talking about RESTaurants. Today: one of Texas's great institutions, Whataburger. ("What a burger!")

Mmm... burgers!

Stay focused. We're talking about State Transfer (the ST in reST). So when I'm driving between offices, I often stop by a Whataburger to get some lunch. When I arrive, I need to know how to order my food.

 GET /order/ HTTP/1.1
 Host: whataburger42.example.com

And Whataburger #42 kindly responds with two links I can follow to make that choice: Drive Through or Dining Room?

 HTTP/1.1 200 OK
Date: Mon, 23 May 2005 22:38:34 GMT
Server: Rosalinda
Content-Type: text/html; charset=UTF-8

<ul>
<li><a href="/order/drivethru/">Drive Thru. Cars in line: 9</a></li>
<li><a href="/order/diningroom/">Dining Room. Cars in parking lot: 14</a></li>
</ul>

It's very likely the cache headers on that response would only be for 30 seconds or so—it's a busy Whataburger at lunch time. So this is classic REST: we're using HTTP to retrieve hyperlinks that navigate us through application state. The hungry client (me!) can choose between two options, and the exact method for specifying which one is simple: I follow a link.

You're a Houstonian. Surely you always take the drive through?

Surely. Well, actually in the above case I'd almost certainly go into the dining room. And theirein lies the interesting lesson of the Whataburger concurrency dilemma. The two order pipelines handle state transfer entirely differently, because of concurrency and correctness issues. I won't keep posting silly HTTP transcripts, but you can play along as if I did. So whether I'm in the dining room or the drive through, I need to order my food. We might imagine that a GET of the resources presented above (e.g. /order/drivethru/) returns a <form> which I can POST to in order to create an order. This works in both places, right?

Actually, sort of. And here we get back to the concept of getting it right. If I'm ordering food during the incredibly busy lunch hour, my order goes through a canonicalization. If it's 3.30 in the afternoon, the B-team is on staff and just takes my order and gives me a number. Why? Let's imagine I'm in the dining room and order (POST) something like "Um, I'll take a #1 meal, with onion rings instead of fries, and a drink. Oh, no pickles." At lunch time they can't afford to screw up their pipeline of burgers, so they'll ask some clarification. "Do you want cheese with that?"

Of course, everyone loves cheese.

What does that look like in a REST universe? The restaurant doesn't even need to issue me an order number before they come back with their upsell/correction. So I imagine they'd simply return yet another <form> which I'd need to fill out. Perhaps a <form> to fill out with restricted options: the order you gave me, or the order you gave me with cheese. If I'm in the drivethrough, they always canonicalize the order in a certain form to absolutely minimize confusion. My order becomes a "#1 w/cheese, no pickles, onion rings, diet coke". That's another representation of the same resource (my order), but the server is insisting on canonicaliation because at lunch time getting it wrong is too expensive.

Got it. I'll be ready the very moment cars come standard with an HTTP client.

Don't be a smart ass. This simple transaction (and we're not even done yet!) already elucidates one example of choices for managing state. We POSTed to a URL which gave us another form to interact with. In the real world, the state of that particular application involves Rosalinda—our ever cheerful cashier—and I remembering what we're talking about. If we came back in an hour, we'd have to start at the "Um". But in our REST example, the state of the conversation is entirely contained in the form I got back asking me if I wanted cheese with that. Rosalinda doesn't need to give me an order number or alter her databases.

When does this become relevant to my day job?

Lots of transactions need canonicalization. Geocoding is a great example. We've all typed "800 8^th St" into a mapping program and had it answer not with the map we expected, but a form or list of hyperlinks asking whether we meant to ask for that address in Port Arthur, Hempstead or Port Neches. (It could even be an "HTTP 300: Multiple Choices" response, but something tells me that fine a reading of the HTTP spec is some years away.) Those links contain the entire correct canonicalized address, and the server doesn't need to remember it was talking to you. My request for 800 8^th St Port Arthur is indistinguishable from the less knuckleheaded person's request who asked for it correctly in the first place. I arrived at the same application state. Yes, I wanted cheese.

That's a good point, you said the drive through and the dining room were different, but all we've talked about is identical canonicalization processes.

Well, both order channels have guided me through their state identically so far. In the dining room, my final POST of a canonical order results in the creation of an order resource: I get a little orange plastic order number: 23. In the REST world, I am told my order now exists at /order/diningroom/23. I can GET the status of that resource as often as I like. Is my burger ready yet? Is my burger ready yet?

Rosalinda's co-worker Randall is happy to tell me as often as I ask that my order is or isn't ready yet. But he tells me immediately. And he's also answering my fellow hungry diner's queries as well: #21, #22, and #24 are all asking. In the dining room, food is served asynchronously. When it's ready, it's ready. It would not be unusual to get my order before #21 if he also ordered a milkshake and biscuits. One of these times, my GET will results in a beautiful (digitally signed!) cheeseburger. (And this is why you choose the dining room over the drive through when the drive through is long. You can get your food in the average time it takes to prepare it under load, not the sum of times it takes the people ahead of you to fill their orders.) After my digitally signed burger is eaten, if I GET at the same resource again, I will probably be told "HTTP 410: Gone" or "HTTP 404: Not found". (Yes, I'm totally ignoring security and the possibility someone will steal my burger by guessing my order number. There are many orthogonal ways to handle that.)

Back in the drive through, the state transfer is totally different. A simple order number and asynchronous handling is not enough. Cars must be served in order. I have to wait in line. My POST to create a new resource won't return a nice URL I can poll on. It will probably block until it returns the digitally signed cheeseburger. I get on a busy web server and had to wait behind other requests. Where was the state? It was all in the server: shuffling connections, building & servicing queues, etc. As far as I was concerned, the application was stateless.

But at what cost?! The server (the drive through) had to maintain an open connection with me that whole time. And remember what my order was. Heavy duty, man. And it's not very scalable. In the dining room, Randall could easily handle dozens of diners asking him where is order was. The diners held onto their own state—he hardly had to remember anything! But in the drive through, cars are waiting in line and waiting in line and my dreams of a fast lunch are shattered when I see the car in front of me ordering 12 burgers for her office lunch.

In return for the simplicity of simply POSTing a blocking call (easier to program—you can leave the air conditioning on), the server takes on a heavy burden. The Whataburger near my office chooses an alternative to asynchronicity in an attempt to scale: they have two drive-through lanes. When I POST my order, I am probably getting an "HTTP 302: Found" or "HTTP 303: See Other" telling me which drive through URL to make my blocking post to (e.g. /order/drivethrough/1).

Now I'm hungry

Not me. I got today's burger at the neighborhood beer joint a few hours ago. They handle their scalability and state transfer issues like Whataburger's dining room: my number today was 8.

What Can a RESTaurant Teach You About REST? (part 1)

noreply@blogger.com (Sebastian Good) — Wed, 15 Aug 2007 04:29:00 +0000

I'm buying a house soon. That's the sort of transaction you have to get right. No matter what the cool kids are saying about transactions being dead, I'll fight Brewer's conjecture all the way the bank. I want my house, and the seller wants his money. It's definitely not okay for one of us to end up with both. Why? Because it's too hard to correct the error[*]. Therefore we pay the not inconsiderable overhead of title companies, escrow agents, loan officers, wire transfer fees, etc. to get it right. It's way cheaper than going to court.

But in the rarefied world of blogosphere REST pundits, we evangelize the webby way for lots of things. Fire and forget. Assume your communication channel is going to fail a lot. Assume statelessness. This would totally suck for buying a house. Imagine having to bring all 300 pages of documentation required to every meeting you attended—tax returns, site surveys, credit reports. And how could anyone be sure you were being consistent? Boy I would love to have brought a different set of papers to the loan officer as I did to the IRS. (Hello, sub-prime mortgage crisis! But I digress.)

But aren't we talking about RESTaurants? I thought it was a clever pun.

Sorry, yes we were. But the point was to introduce the cost of getting it right. And besides, it's hard to concentrate on anything else when you're buying a house, so indulge me.

What, honestly, is the cost to you if a restaurant fails to get your order right? Mistakes are made all the time by waiters, customers, busboys, managers, and cooks. Yet, unlike closing on a house, you do not sign contracts or involve lawyers when you order food at a restaurant. In fact, you don't even do a credit check; centuries of social convention have blessed us with a system where it is assumed you can pay for dinner and only have to prove it at the end.

Or has it? I once booked an anniversary party for my company at a tony restaurant in Las Vegas. I had to put down a credit card to hold the table. Why there and not at, say, my neighborhood pizza joint? The answer is fairly obvious: if I flake on my big party at said tony restaurant, they're out a private room and four figures. If I flake on dinner with my wife at said pizza joint, they'll fill the table anyway most nights, and if they don't, they're out 20 bucks, tops.

Okay, now you're bugging me. REST pundits are supposed to wax poetically about URL design and resource representations. You know, eBay transactions are really resources, what is the URL of a pixel[**]?—that sort of thing. This is a rambling diatribe about transactions, not REST. You can't fool me! Though you are at last talking about restaurants.

Well, REpresentations are only half of REST. (As measured by the letters they get in the acronym.) State Transfer is pretty damn important in a stateless protocol. Transactions are only one kind of state transfer. And what I'm warming up to talk about (warming up, get it? restaurants? anyone? is this thing on?) is state transfer. How does that latte get to you at Starbucks? My jalapeño sausage at the local barbecue joint? My #1 Meal, cheese, no pickles, onion rings and a Diet Coke at Whataburger? My Kansas City Strip at Delmonico? Getting there involves many state transfers, and each of these restaurants has chosen a different system.

But now you've gone and spent all your time on silly jokes.

So I have. See you tomorrow.

[*] Oh yes, funny story. When I bought my first house, the combination of my naiveté and an under-trained clerk at the title office led to me bringing a personal check for the down payment on my house. It was an average house, but a 20% down payment still made the check five figures. The title company protested that they couldn't be expected to float that kind of money waiting for my check to clear. And besides, we had already signed so much paperwork and gotten everyone in the same room, that it seemed unlikely we would want to do this again in 5 days. I said, "The check won't bounce, and if it does, what's the problem? You know where I live, right?" Cursory examination of the 300 pages of paper signed that afternoon indicated more than a few copies of my old and new addresses. Some uneasy laughter ensued, and that was the end of that. I got lucky. That actually would have been a very expensive error on the title company's part.

[**] Turns out, whole pictures can have URLs. That's an approach I'm pretty sure wouldn't work for satellite imagery. But again, I digress.

The Mac is Back

noreply@blogger.com (Sebastian Good) — Tue, 14 Aug 2007 23:49:00 +0000

Tiny post, for anyone else wondering about high CPU utilization and slow or crashing network services on their Mac while running Parallels Desktop for the Mac 3.0. I've been a happy Parallels-er, running a few critical apps on a virtual PC while doing most everything else on my Mini Mac. (Not a speed-demon, but quiet, cool, and dual core.) After the upgrade to 3.0, I found Parallels was routinely sitting on 40% of my CPU, and causing network connectivity, especially in Safari, to totally suck. I had to Force Quit Safari daily. That didn't make sense. A few other people seemed to suggest that file sharing might be the problem. I've disabled file sharing of the Windows folders into the Mac. This is a nice feature, but in practice I don't use it (or the reverse) terribly often. All my permanent files are on a central networked drive which serves both Macs and PCs in my house. Disabled the sharing, and all was better. Perhaps they'll fix this in their next patch and let me use more than 1 CPU in my virtual machine, or I'll have to join the hordes defecting to the newly rich VMWare.

Added 19 Aug 2007: It appears USB support is no picnic for Parallels either. By disabling USB devices when I'm not using them in the VM, CPU usage has dropped quite a bit also. It still sits at 13% when idle, which I find disturbing. Experiments with VMWare are ongoing.

REST for Rasters: a RANGE of options

noreply@blogger.com (Sebastian Good) — Fri, 10 Aug 2007 21:12:00 +0000

Hey, I know a dogpile when I see one. I'll jump in. (Seriously, follow that link first, then read the response.)

Sean's spot on. Resources can have subcomponents. I even—while addressing a different but related point—have opined about file-based formats not forming a sensible basis of REST rasters. Dividing rasters into interesting slices as Sean has suggested is the right big idea. As far as suggestions, I've been meaning to write about the following because we've been working on them on one of my projects.

RANGEd (partial) GETs. HTTP GET supports ranged requests. In other words, I can ask for specific bytes of a file. In this case, a smart raster client could indeed notice that a resource was a GIF, and only ask for the first few hundred bytes, read critical header information, and use random access to obtain the actual pixel values. A naive approach would end up being awfully chatty, and it leaves the resource a bit more opaque than you might like (after all, the byte range is not part of the URL) but it is a practical solution.

Standard Raster Accessors. Libraries like GDAL or ESRI make very very similar assumptions when dealing with raster data. Everyone agrees that rasters have bands of data, might hold data of different types (ints, floats, bits, etc.), are conveniently arranged in tiles, etc. It is probably not too hard to agree on a standard header for raster information to be returned to a HEAD request. And then a fragment/anchor identifier language could be used to agree on raster chunks to be returned in an appropriate binary format. (That's the part after the '#' in the resource.)

http://example.org/seafloor.png. The whole shootin' match. Ask for a HEAD or do smart partial GET requests.
http://example.org/seafloor.png#*.0,0.100,100. Gets all bands (*) and all pixels between (0,0) and (100,100) pixel coordinates.

The actual syntax above is obviously only a stab in the dark, and perhaps there are already better examples in things like the OGC Web Coverage Service, but the fact is indexing into even very large rasters doesn't take a lot of data, and it'd fit nicely in the fragment/anchor. Which would be very RESTy, very URL chic.

spatialreference.org: Let the Sun Shine!

noreply@blogger.com (Sebastian Good) — Thu, 09 Aug 2007 05:59:00 +0000

Every once in a while someone spends the weekend finishing something that should have just been taken care of years ago. A few weekends ago, Howard Butler and Cristopher Schmidt did that with spatialreference.org. Someone needed to take the EPSG spatial reference database and web-enable it. OGC's solution was to demand a solution so obscured by me-too XML-ism, it could only be built over many many years by their favorite XML promulgators, Galdos. And in fact, it's still not there. Howard & Christopher realized what Steve Jones pointed out: in 2007, CRUD applications should be considered "dull, boring and uninteresting". Yes, it should be possible to CRUD-ify the EPSG database in a matter of hours and get on to the interesting stuff. Thank goodness they did.

Yes, they've taken the latest EPSG codes and made them available for download and upload. But hey, here's the extra cool stuff they can do now this CRUD's out of the way and trivially accessible via an easy to program REST service.

Get spatial references in a variety of formats! Yes, it's all well and good you & I agree we are talking about my favorite coordinate system, Aratu / UTM Zone 24S. But I love the fact that by just asking for the same URL postfixed by "ESRIWKT" I can serve my corporate overlords by downloading a PRJ file so I can do the math with the ESRI projection engine. Or by postfixing with "proj4", I can work instead with everyone's favorite open source engine. That's value add.
Those usage rectangles hiding in the EPSG database can be used to tell me where that projection is preferred or valid! No sooner did I predict to some colleagues that these crafty Python programmers would find it easy to make the usage areas available (Tuesday) than those crafty Python programmers just did it (Thursday!), complete with nifty map showing valid areas! Yep, Aratu is used off the coast of Brazil.
I can search for different coordinate systems without having a copy of MS Access on my machine.

And because they're programming simple REST in Python, they can churn out features in hours instead of months. Let's hear it for spatialreference.org.

I am sure that spatialreference.org will be seen as a dangerous influence. I have already seen stiff resistance to the concept. As a GIS programmer stitching lots of systems together, I've been waiting years for a standard authority that's actually user editable and supports multiple formats. So please forgive the following rant. Here's what the geodetic elites will say: We can't have amateurs doing this, can we? How can we be sure these crazy Python people aren't secretly corrupting the EPSG database they're supplied with by, um, EPSG? How can we be sure they're even keeping it up to date and avoiding errors? (GDAL's enormous world of contributers and commercial support apparently not withstanding. Sorry, Frank, I guess.) We recommend not using it; you must wait for the OGC/Galdos officially sanctioned SOAP monster from hell this September!. What if all the EPSG database needed was a better technical solution from people with technical common sense, instead of years of hand-wringing by otherwise very smart people? Look, these standards improve because they're widely used. Locking up EPSG codes in an Access database begs them to be copied offline, abused and misused. Opening them up to the harsh daylight of global interoperability will get them cleaned up in a jiffy. Sunlight is the best disinfectant.

Things I'd love to help patch into this beauty:

Units of measure. The EPSG database defines a wealth of linear and angular units of measure, but a nice user-defined database of other units would sure be handy. Degrees Celsius per millisecond, anyone?
Coordinate Transformations. These are a bug-bear for everyone. Hyperlinked lists of transforms appropriate for different projections and areas of the world are sorely needed.
Secure Endorsements & Selections. Companies often invent new reference systems and transforms. They should be able to upload them and sign them cryptographically so that on download they can be sure they have not been altered. EPSG might even sign their own with a public certificate as part of a formal review. Also, many users may wish to browse the database in a way that limits their view to only common systems in use in their area. A Brazilian oil company is going to be endlessly fascinated by Aratu and the Illustrious South American Datum 1969, but annoyed if they have to wade through four screens worth of Xian 1980 / 3-degree Gauss-Kruger zones. Why not let that oil company create their own lists of preferred systems (and transforms, don't forget the transforms!) and sign those lists? Coolness all around.
More Authority Translations. What is the Blue Marble factory code for Aratu? Mentor? These take a bit of input from the companies themselves and the geodetically minded. spatialreference.org merely provides the clearing house and brings the discussion out into the open where it can be vetted.
Warnings of Incomplete Translations. Aratu is one of my favorite examples because it often highlights differences in projection engines. ESRI knows that the Aratu datum is based on the International 1924 ellipsoid. But it's a datum, not a pure ellipsoid. PROJ4 doesn't know Aratu, it just calls it "+intl". The PROJ4 codes given by spatialreference.org (by way of GDAL, if I understand correctly) for "Aratu" and "Unknown datum based upon the International 1924 ellipsoid" are the same ("+proj=longlat +ellps=intl +no_defs"). Yes, the latter is marked clearly "Not recommended", but the Aratu page's PROJ4 version merrily drops the Aratu-ness right out of Aratu. Icky. Again, this isn't an issue the Python boys can merrily solve; it's a thorny data issue. But by providing a framework for that to be annotated, good things can happen.

Let the Sun Shine!

Google to US Government: Open The Airwaves Or We'll Buy Them. All of Them.

noreply@blogger.com (Sebastian Good) — Fri, 20 Jul 2007 18:43:00 +0000

Google demanded today that the US Government agree that use of the public wireless spectrum be subject to the following four rules, meant to require the airwaves be "open".

Open applications: consumers should be able to download and utilize any software applications, content, or services they desire;
Open devices: consumers should be able to utilize their handheld communications device with whatever wireless network they prefer;
Open services: third parties (resellers) should be able to acquire wireless services from a 700 MHz licensee on a wholesale basis, based on reasonably nondiscriminatory commercial terms; and
Open networks: third parties (like Internet service providers) should be able to interconnect at any technically feasible point in a 700 MHz licensee's wireless network.

Fair enough. Google isn't a phone company, is sad that the phone and cable companies want to charge for their networks, so they want Uncle Sam to demand that Google be allowed to play with the big boys. Since Uncle Sam granted the monopolies the phone and cable companies are abusing, I say this is all good.

But the kicker, and it's an amazing kicker, is that Google is willing to pay the government $4.6 billion if they agree to its terms.

That's why our CEO Eric Schmidt today sent a letter to FCC Chairman Kevin Martin, saying that, should the FCC adopt all four license conditions requested above, Google intends to commit at least $4.6 billion to bidding for spectrum in the upcoming 700 Mhz auction.

Why $4.6 billion? While we think that a robust and competitive auction based on these four principles will likely produce much higher bids, and we are eager to see a diverse set of bidders competing, $4.6 billion is the reserve price that FCC has proposed for the auction. With any concerns about revenue to the U.S. Treasury being satisfied, we hope the FCC can return its attention to adopting openness principles for the benefit of consumers.

I think the obvious question is as follows: If Google is willing to spend $4.6b to encourage the government to adopt its standards, what is it willing to spend to buy the whole spectrum and force its way if the government disagrees? And will it—can it—outspend the phone companies to get its way? And if they do get their way, will they still "do no evil?"

It's a bold move. That's a lot of money, even for Google. But it doesn't seem quite subtle enough to work. Would the commissioners FCC rather do the "right" thing under such naked, public threats by such an arrogant company, or slide a deal in the backdoor while receiving some nice golfing trips, fine meals, cash in a freezer, and evening entertainment?