Largely harmless confabulation on commercial software development, economics, and the relentless desire to see more of the world
This evening was a bit of a novelty. Forgoing the big ESRI party, we had a nice dinner on the shore near the USS Midway. It turns out another conference in town rented the venerable old carrier and flew the Goo Goo dolls in via black helicopter to play a full concert. The park near the boat was filled with people on chairs & blankets enjoying the free concert. Pretty cool.
As they did at the Developer Summit, ESRI gave a talk a talk today on SQL in the Geodatabase. I left equally as excited from this talk as I did from the very similar talk at the Developer Summit. In that time, Microsoft has declared it will launch its own spatial type, and ESRI has clarified that it will be supporting it. And on the the PostgreSQL front, they will certainly be supporting the PostGIS spatial format, as well as their own implementation of ST_GEOMETRY on that platform. These developments, plus new 9.3 developments discussed today in the usual "road ahead" slides, are wonderful.
It has always seemed that "GIS" departments were about as silly as "MS Word" departments. GIS implies certain kinds of complexities—spatial references, spherical geometry, topology calculations—which certainly require expert guidance, but from an IT perspective something is wrong if doing GIS right means being forced to use systems which insist on being at the center of the universe. If we are dealing with document management, we recognize that while a particular word processing package (e.g. MS Word) aids us greatly in formatting, composing, editing and comparing individual documents, it shouldn't demand to archive and version all documents of interest in my company. After all, what about the pictures, faxes, well logs and credit card receipts? Similarly, just because a GIS data set should be in a database doesn't mean that I should be forced to go through the GIS system to talk to that database.
That is to say, a shape (or raster, or topology rule, etc.) shouldn't really be any more special than other data types in a database. (There is no VARCHAR department at any company that I'm aware of.) For the longest time, dealing with shapes as equal citizens wasn't practical: there was no widespread agreement on shape representations in a database. But now there is: the OGC ST_GEOMETRY specs, and it's wonderful to see ESRI implement it. Finally our shapes can be attributes, subject to the wonders of SQL and all the millions of tools out there for data processing. This is the direction GIS should be going. Scheduling software can read ST_GEOMETRY to understand location, not that you have to buy a GIS scheduling package just because you want to be location aware.
And for the happy conclusions from today...
...Confirmation that SDE is happy to deal with ST_GEOMETRY type SQL queries neutrally. You can do things ArcObjects doesn't like, like have multiple geometry types in a single column, or put multiple shape columns in a table. You can deal with that complexity as you like in SQL, and expose simplified views to ArcGIS. One presenter suggested that even restrictions like those were being reconsidered in the "post-9.3" nirvana. It will be a profound change: the shape becomes a simple attribute, amneable to simple analysis.
...Oracle's default geometry type in 9.3 will become ST_GEOMETRY. This is because Oracle is deprecating the LONG RAW on which SDE BINARY relies in Oracle 11. The SDE presenters were studiedly neutral on which of SDO_GEOMETRY or ST_GEOMETRY was the "better" choice, and I'll depart character for once and not speculate on that. But given that ST_GEOMETRY will be a lingua franca in several databases and appeared (in at least one slide) to be a more compact representation, that'd be my choice. (In today's network-bound world, it seems storage size matters more and more.)
... SDE 9.3 will assume Unicode (UTF-16) for all new feature classes unless told otherwise. That's sure to throw some for a loop, but it's also the right thing to do.
The slides from the talk will certainly contain much more detail at a better level of accuracy, so keep a sharp eye out. You know whenever Gudmundur Hafberg shows up at a talk it will be absolutely crammed with incredibly useful information backed by hard metrics.
And with that the UC is over. It's a shame to leave the beautiful sunshine here in San Diego, but 4 days in this cult, friendly though it is, is probably enough.
In yesterday's post on fun new ArcGIS Explorer features, I mentioned that I was optimistic AGX's (and other's) bugs would be fixed because of new "positive pressure". The keynote for the 2007 ESRI User Conference indicated that ESRI's internal bug-database would be opened up for public viewing "soon". This is an incredibly gutsy move on ESRI's part, given the presumably very large corpus of existing bugs out there. That said, as a developer I can tell you a great number of application bugs are never reported to the development team simply because (a) users assume developers already know about the bugs or (b) it is hard to send useful bug reproductions. I hope that the open database reveals both to ESRI management what bugs are most "popular" in a much more egalitarian way and that ESRI users are encouraged to actually help developers out.
We regularly report ESRI bugs to the regular support systems, and since roughly 2006, have gotten excellent repsonses from their staff. Yes, we see few of the bugs getting fixed, but beeing able to see progress of bugs, posted workarounds, etc. in a more direct format will be really cool.
I see that there has been a post about it from the ESRI elves. Nothing focuses everyone like clear descriptions of the problems for the community to discuss. This is a truly golden opportunity, and if it survives the year, might make an amazing difference in ESRI's software quality, which would be reason to cheer.
At the 2007 ESRI UC today, we heard cool news at the ArcGIS Explorer sessions. The next rev of AGX (they are aiming for a release every 12 weeks) will contain several look-small-but-act-big improvements and one fairly-big improvement. And I put in another vote for multi-threaded tile retrieval, but I'm sure they're tired of hearing about it. Thanks to the always cheerful and friendly ESRI developers at the ArcGIS Explorer island.
Apparently an internal AGX once included a 2-D map viewer as well as the globe control it's got now. For global datasets, a map is a better option than a globe which hides half the planet. However it was apparently scrapped because the navigation paradigms are too dissimilar. This makes some bit of sense. The "post 9.3" nirvana is when they are working on a combined 2-D and 3-D rendering engine, and that's when we might see a map control in AGX. I'm sympathetic to the problem—these renderers grew up on their own and have all sorts of organic cruft on them making their merger difficult. Furthermore, once you're 2-D you're in the confusing world (for most end-users, certainly) of coordinate systems. Given the sex-appeal of the globe and the genuine usefulness of 3-D scenes, I can't imagine this being an issue in the public market. But I'd sure like it.
So now I've gone and raved about upcoming features when I just got finished complaining that ESRI was, in general, introducting too many new features at the expense of fixing bugs. For AGX, I think their 12-week cycle puts them at an advantage. As always, the developers at the conference gave excellent, well-practiced presentations and were deeply knowledgeable when asked questions. I hope ESRI's opening up the bug database will provide the sort of positive pressure their management probably needs to let them fix more bugs.
Well, I'm at the ESRI 2007 User Conference for the first time in three years. There's apparently 13,500 people here — there were "only" 10,000 last time. The San Diego conference center handles them all gracefully.
The highlight of the first day was, without a doubt, the forceful and intelligent presentation by The Green Belt Movement's founder, Wangari Maathai. Her discussion of poverty and environmentalism was moving. The GIS demo they managed to wedge into her talk was distracting, but mercifully brief. After hearing the ESRI propganda machine in full roar it's easy to be cycnical about Jack Dangermond, but his dedication to conservation and the environment is genuine. Seeing him highlight social justice and conservation is good.
That aside, the plenary was quite disappointing. While some of the usual suspects have round-ups that make it look like a lot is being added, I find little to be energized about. They are integrating Image Server with ArcGIS server. Wonderful, I can't wait to see how badly hacked together that API will be. They are enabling "mash ups". Welcome to 2005. Thanks. [Update: Breathless cynicism about PostgreSQL support has been removed from this post.] They admitted that their support stinks, but as Dave Bouwman properly points out, their spin ("we were overwhelmed by demand") is hogwash, no matter how pretty their info-porn chart was. Support is overwhelmed because the product has more bugs than a sewer tunnel in summer. Yes, good of you to apologize in front of 13,000 people. Seriously, that's something. But then it was the usual schtick: hours of demos of new features. New features are directly correlated with more bugs, not fewer bugs. Simple as that. It is abundantly clear ESRI is not serious about reliability or usability. (One lame "pause labeling" button notwithstanding.)
They're not serious about speed either. Their ArcGIS Explorer demo was "screaming fast". Well, sure it was. They must have had that blessed thing caching for a day. Dave, go with your gut. Of course it was all cached. Have you ever watched it in real life?
Dangermond's discussions of Google, Yahoo and Microsoft seemed just as ginger as they were at the Developer Summit. He admitted they were broadening the reach of GIS to the public, which fits nicely into his "I'm just in it for the science, what I'm a billionaire too? " spiel. Of course he must be terrified by companies who make more in a few months than he has in his whole life. The attitude at the moment seems to be, "If you're using Google or Microsoft for anything, you're clearly an amateur, and you can go $*#& yourself." Of course Grandpa Dangermond would never say that — he let his attack dogs do it on stage, but the effect is the same.
True "professionals" will make "authoritative" data and "publish" it to these "free, consumer" services. Now don't get me wrong: the authoritative legal boundaries for the lot my house shouldn't be created by a bunch of click-happy MySpace kids through Google Earth. True. But dismissing the entire consumer-based renaissance (and simplification) of GIS strikes me as a bit silly. Like Microsoft, ESRI has no choice but to continue focusing on adding more features to win ever more select enterprise deals, defending their monopoly. They are too hidebound by backwards compatibility and their own elitist echo chamber to participate in the mass revolution.
You know what, if I ran ESRI I might do the same thing. After all, running further and further up the value chain makes money. Building applications the rest of the industry has had for decades (e.g. the much-lauded "ArcLogistics" demo) and calling them new GIS software must be satisfying as a programmer. (Seriously. Scheduling? This is a new problem?) But as an attendee of the show, and especially as a programmer, I don't see the wonder and sparkle in it. I see endless corporate sales drivel and really really pretty graphics.
Let's hope the presentations of the real software reveal some more satisfying detail underneath this depressing propaganda. And thank goodness for the wonderful presentation by Dr. Maathai.
