Help Help Needs Needs Explorer Explorer ArcGIS ArcGIS: HTTP Adventures

It was with great enthusiasm that I fired up ArcGIS explorer a few weeks ago to kick the tires. The spinning-globe phenom shows no signs of abating, and since ESRI is the tool of choice at big companies, people like me have to know how to put data on ESRI's spinning globe. In this regard I am hardly unique. However I've never seen an analysis of ArcGIS Explorer's network behavior. It's interesting.

For various reasons, we want to expose some information to ArcGIS explorer as WMS services. (Perhaps I'll expand on these in future.) So in order to see what was going on (and if we could get in the way to do anything clever), we immediately fired up the tool that should be at the top of every web developer's list: Fiddler. Fiddler acts as an HTTP proxy, allowing you to spy on every action an HTTP client takes and spy on every response its servers send back. The results are fascinating.

The first trick is always to get Fiddler to step in the middle of this HTTP traffic. Strangely, ArcGIS explorer doesn't use WinINET proxy settings (the ones you set from IE's options panel and that most Windows apps share.) Since ArcGIS Explorer runs on Windows, I'm not clear why the developers skipped this. It's a vague annoyance, but at least the proxy settings can be easily set.

Next we watch the traffic. I pointed at one of the pre-canned WMS services ArcGIS Explorer ships with (map.ngdc.noaa.gov) and watched the data come in.

The Tiles Roll In

As expected, ArcGIS explorer asks for maps a tile at a time, and the tile sizes are nicely predictable, generally chopping the world in half as it goes. Here is a typical (formatted) GET tile request.

GET /servlet/com.esri.wms.Esrimap ?VERSION=1.1.0 &REQUEST=GetMap &SRS=EPSG:4326 &BBOX=-90,-45,.000000000000000,45 &WIDTH=512&HEIGHT=512 &LAYERS=WorldGrid,Countries,Cities &STYLES= &EXCEPTIONS=application/vnd.ogc.se_xml &FORMAT=image/png &BGCOLOR=0xFFFFFF &TRANSPARENT=TRUE HTTP/1.1

As said, the good news is that the tiles are regularly sized (512 pixels) and on predictable boundaries (even fractions of 90 degrees). They're being requested as PNGs with transparency, and perhaps most importantly they're being retrieved as GET requests, so they are cacheable. And now for the not-so-good news, which somewhat overwhelms the good news.

?Backwards Requested Tiles The Are Why

Well, if you actually look at the tiles in the order they are requested, you invariably notice that Explorer asks for them in the wrong order. Here is what my final globe looked like:

And here is the order I got the tiles in.

This is close to what I want.

Fair enough, it's most of my field of vision, too.

Wow. Where did this come from? It's not even visible on this globe and the side I'm looking at isn't finished!

Dude, Alaska is barely visible on this globe and I haven't seen Canada yet!

Okay, there's Canada. Where's Mexico and the Western US?

Greenland - good job.

More Greenland.

Uh oh. The northern coastline of Russia. Not visible on the globe, and we're getting down to smaller detailed tiles, when we still haven't received the full tile set at the larger level. I still have blank spots on my globe.

Europe, good. At least we're not in Russia anymore.

More Russia. Can't see it.

And more detailed Russia. Where the heck are Mexico and California? We're still waiting.

Awesome. Antarctica — not visible.

More Antarctica. Perhaps ESRI developers liked Happy Feet?

... and we skip another 6 tiles with beautiful detail of the coastlines of Chile and Argentina (both invisible on this view) ... until we finally get to the very last tile obtained:

Mexico and California!

Oh, Actually the Tiles Are Different Sizes

If you look at the tiles above carefully, you'll notice they're not all actually the same size. Some are 512x512. Others are 255x256. Another one is 256x257. Why? Rounding errors? Don't know. This isn't really a particularly big issue unless you're trying to pre-generate WMS tiles for clients like ArcGIS explorer... and that turns out to be a bit of a problem since ArcGIS has funny caching behavior.

Could You Say That Again?

It is a real shame that the tiles are retrieved in what appears to be a silly order. But at least they are retrieved as GET requests and they're eminently cacheable by the scalable Internet infrastructure. Surely NOAA isn't actually having to generate real pixels for these tiles most of the time. Wrong. The GET requests indicate no willingness to cache results — ArcGIS Explorer never lets HTTP proxies cache data. This means that the hopes of letting the HTTP proxies of the world unite to lighten your server load, you instead need to do all your own caching or pregeneration.

GET HTTP/1.1 User-Agent: ArcMap Service Layers Host: map.ngdc.noaa.gov Proxy-Connection: Keep-Alive

We know that ArcGIS Explorer caches copies of this data on disk somewhere. Perhaps their strategy was to not clutter people's disks with two copies of the data — one in their Internet cache (built into Windows Vista, for instance), and another in their ArcGIS Explorer cache. This would be the positive reading. But another disturbing fact suggests an alternate reading.

Let's Not Cache It and Ask For It Twice

Yes, ArcGIS Explorer asks for every tile twice. Here is a section of Fiddler's recording of the above session. The left column is the request number, i.e. the order in which ArcGIS Explorer requested tiles. The right column is the size of the response. You can see each request was repeated twice, getting the same result each time (of course).

Multithreading? Almost.

This one is a little harder to demonstrate with screenshots, but observations prove a couple of things conclusively.

• One HTTP connection per server. Despite the HTTP spec allowing each client to make two simultaneous connections to each HTTP server, Explorer doesn't bother. It's a shame, since it'd usually get the tiles to my spinny globe twice as fast. Everyone else does this. Why not? Because...
• One HTTP retrieval thread period!. Even if you connect to two services at once (WMS, IMS, Globe Service, whatever), Explorer will only hold one connection open at a time. So your fastest services are held hostage to your slowest ones. I connected to a WMS which took about 30 seconds to generate an image. This meant that upon zooming to a reasonable level of detail instead of getting a map within a minute or two, it took upwards of 10 minutes because it got tiles in a bad order, got them more than once, and all the while I couldn't see any other context from the other layers because they were held up behind the slowpoke. This is a basic starvation issue I'm surprised the developers haven't considered.
• UI and retrieval thread confusion. If Explorer is waiting on a long-running HTTP session, you can't shut it down. The UI thread apparently blocks on the retrieval thread waiting to shut down. Is this nasty COM STA mojo? Threading can be difficult, especially if you're working in C++. But writing multithreaded HTTP clients is pretty close to being a solved problem...

Forget Wet Paint... We're Still Framing the House

ArcGIS Explorer is not even fairly called a beta product. At best it is a developer's demo or a throwaway prototype. Put together some lessons learned, and let the programmers who have built a real Internet application at the thing. Until then, it'll be an awkward subset of ArcGlobe wandering down the information superhighway with a blindfold and a marked hobble.