Jon's Radio

Nic Harcourt and Elvis Costello contemplate the last flick of the dinosaur's tail

2005-01-27T11:07:08-05:00

Along with driving, hiking, and jogging, snow shoveling is an activity that benefits from time-shifted audio content. Here in New England we've had plenty of opportunities lately to indulge in that least favorite of winter sports. A couple of storms ago I had a chance to catch up with Kent Beck's excellent keynote at the Developer Testing Forum. Yesterday I wasn't planning to absorb ideas, just music. But my selection -- this week's Sounds Eclectic from KCRW's Nic Harcourt -- brought ideas to my headphones as well as Elvis Costello's hour-long studio performance.

This was a departure from the usual format. Normally the studio performances run only about twenty minutes, interrupted by a short interview with the band that, to be honest, I usually skip. But this week Harcourt and Costello let their hair down, talking at length and with complete candor about the new realities of the music business. "It's over, it's done," Costello says at one point. "We're seeing the last flicking of the dinosaur's tail."

The conversation wasn't just about the technology and culture of remixing. It wandered around a lot. Ironically and appropriately, remixing is what enables me to focus on that theme in the conversation and share it with you. Here's a Real playlist that splices five segments into a 4.5-minute collage. And here's an MP3 version of the same thing.

Let's take a look at that playlist file:

rtsp://go.rbn.com/ ... sc050123Elvis_Costello__The_.rm?start=24:26&end=24:35
rtsp://go.rbn.com/ ... sc050123Elvis_Costello__The_.rm?start=21:45&end=22:21
rtsp://go.rbn.com/ ... sc050123Elvis_Costello__The_.rm?start=30:14&end=31:46
rtsp://go.rbn.com/ ... sc050123Elvis_Costello__The_.rm?start=32:48&end=33:47
rtsp://go.rbn.com/ ... sc050123Elvis_Costello__The_.rm?start=40:37&end=41:51

It's just a list of URLs. The only tricky thing here -- and it certainly is an obstacle that needs to be overcome -- is the ease with which the ins and outs can be specified.

My shareable remixed version of the conversation is just another URL:

http://weblog.infoworld.com/udell/gems/nicElvis.m3u

It's amazing that this is possible. It's even more amazing that, although a streaming server facilitates this kind of in-situ editing, you can do the same thing with static content. As more and more interviews and conference sessions produce online audio content, remixing will help us compress time and focus attention.

Blogosphere subscription trends

2005-01-26T12:50:37-05:00

We've all seen the hockey-stick curve that shows the blogosphere growing like gangbusters. But I haven't seen much on subscription trends, so I took a look at the public information available in Bloglines. For a given feed you can ask Bloglines to show not just the count of subscribers, but also -- for the "public" subscribers who allow this information to be shown -- their usernames and the dates when they began subscribing.

Bloglines doesn't offer an API for this kind of data mining. So I used a small Python script to fetch subscriber histories for a selection of tech blogs that I read, and render them as XML. From there, I used Excel 2003 to massage the XML data into tables and charts. Here's one view of the results:

And here's another:

I'm really not sure how to interpret these. The first chart, for example, suggests that while growth is constant -- that is, everyone adds subscribers every month -- the rate of growth varies episodically. It rises steadily through last spring, falls off through June, rises again through September, and falls off again to the present. Is this a Bloglines artifact? How much of this is variation in the rate at which subscribers are switching from other feedreaders, and how much is real growth? Maybe the Feedburner folks can help shed some light on these questions.

In case it's not just an artifact, though, here's something to ponder. The first chart is dominated by the seven blogs that added the most subs in December, the last month for which there's complete data. But if you look at the remaining fourteen you can see a leveling or slight uptick in January, and that's even without the last week's worth of data. The second chart breaks these individual trends out more clearly. I showed this to Chad Dickerson and he offered a fascinating speculation. Could we be seeing a combination of long tail and power law effects?

Where was desktop search when we needed it?

2005-01-25T10:07:18-05:00

Desktop search engines are sprouting like weeds. Google's showed up in October, then Microsoft's in December, and now Yahoo's this month.

...

For lots of people I know, any one of these choices will produce a life-changing productivity boost. For me, though, that's no longer true. The Gmail experiment has become a lifestyle choice. I still maintain a local Outlook mail store, and it's indexed several ways, but I rarely need to search it. Similarly, most of the documents I create -- InfoWorld stories, Weblog postings -- are pushed to the cloud and are searchable there.

Few of you can or should live in the network cloud to the extent that I do. But if you refocus on the corporate intranet, cloud-based storage has compelling advantages. The first and most obvious one is the ability to access your stuff anywhere, anytime, from any client. A subtler point is that documents in the cloud are documents that other people can help you tag and find.

The gating factor for desktop search is metadata. Finding documents based on words they contain is a huge benefit, something we should all have been able to take for granted years ago. But the future isn't faster or prettier full-text search; it's more context and better relevance. And your personal hard drive isn't the garden where these flowers will grow.

Google's PageRank showed us that relevance is a collective judgment. Services such as del.icio.us, Flickr, and Furl are likewise showing us that metadata tagging wants to be a group effort. One of the ironies of desktop search may prove to be that, by the time it went mainstream, the personal hard drive was about to become an endangered species. [Full story at InfoWorld.com]

Today's screencast is a 90-second short film that demonstrates Simon Burns' astonishing finding that desktop search has been right under our noses for years -- built into Windows but then crippled by weird design choices. In the screencast you'll see how even when the Windows indexer is enabled, a search won't use it unless you prefix the search term with an exclamation point. You'll also see the ancient and obscure Windows indexing administration tool in use. It's bizarrely named ciadv.msc and tucked away in the system32 directory, but if you point a shortcut there you can bring it to life.

This week's column explains how these odd circumstances arose. Even though it's water under the bridge now, I can't help wondering how the desktop search market might be different today if something like this screencast had been widely available three or four years ago. Audio/visual narration of software behavior makes connections in our brains that don't happen when we just hear about software, or read about it, or see static screenshots.

Several folks commenting on my Wikipedia screencast have said things like: "I've tried to explain to people how Wikipedia works, now I can just point them to the screencast." And that was the real point of the exercise for me. We need a new medium in which to communicate about software. The fact that both my Wikipedia screencast and the vnc2swf page show up today at del.icio.us/popular/ -- the vnc2swf surge owing to Nat Friedman's Beagle demos -- tells me that my efforts to popularize screencasting are bearing fruit.

Coincidentally Beagle, the successor to Nat's Dashboard project, is yet another desktop searcher. The more the merrier. We should all expect to enjoy immediate search of local content; it's nuts that we've gotten this far without it; solutions are long overdue.

In the column I argue that our increasing reliance on network storage makes the need for desktop search less acute than it formerly was. I should add, though, that the core technology of desktop search -- that is, noticing and cataloging information events -- will only grow more relevant with time. Whether or not your personal information lives on your personal disk, and whether or not the index is built and searched locally, the reading and writing and editing and communication events occur on the client. Pervasive high-performance monitoring of that event stream will help us weave local storage and the cloud into a common information space.

Heavy metal umlaut: the movie

2005-01-22T16:49:17-05:00

Today's screencast traces the evolution of Wikipedia's Heavy metal umlaut page. I noticed it when both Tim Bray and David Weinberger pointed to it, but the page actually dates back to April 15, 2003.

It's a wonderfully silly topic, but my point is somewhat serious too. The 8.5-minute screencast turns the change history of this Wiki page into a movie, scrolls forward and backward along the timeline of the document, and follows the development of several motifs. Creating this animated narration of a document's evolution was technically challenging, but I think it suggests interesting possibilities.

Deep structure redux

2005-01-20T10:36:48-05:00

I've been monitoring with great interest the reactions to yesterday's item, to its related InfoWorld column, and to Jonathan Edwards' screencast and paper that I cite in both. I was particularly curious to see what Smalltalk and Lisp folks -- who have in certain important respects been there and done that -- would have to say. James Robertson was merely dismissive in this post. Preston Bannister offers more nuanced reflections:

The example and the accompanying narrative is somewhat interesting. Oddly enough it is not the tree-structure-editor aspect that I find compelling. Rather it is the use of sample values, live computation, and highlighting of "dead" (not executed) code. [Structure editors, IDEs, and another Lisp flashback]

Agreed. That's the part of the demo that makes Edwards' notion of example-driven programming come to life. Bannister goes on to say:

One of the strong, subtle advantages in the Lisp environment was in having the parser and pretty printer as simply another set of functions in the library. In the C/C++ world the compiler is a large monlithic seperate program, and offers only a one-way path from source text to compiled code. You cannot reuse parser or gain access to the parse tree. In the Java world the compiler is once again part of the environment (with only slightly awkward access). This is one prime reason Java IDEs can support refactoring, and "deep" operations on your Java programs.

In my opinion at least, the notion of tree-structure editing is still only of limited value, and not a replacement for simple text editing. Back in college, after we had all played with structure editors for a time, one member of our group identified the source of our discomfort by noting that many simple common operations in a text editor were much more complicated in a structure editor. [Structure editors, IDEs, and another Lisp flashback]

Yes. That's why I wondered in my column whether emacs will ever be pried from my cold dead fingers. Mark Levison, who has been thinking about these issues recently, has some suggestions:

IDE's need to better tools to manipulate their Abstract Syntax Trees

The Abstract Syntax Trees need to be documented and made accessible to external vendors
[Mark Levison: Programs still edited as text"]

In the .NET world there is CodeDOM -- though I gather it's more for creating than for analyzing and rearranging structure. For Java I found the AST View plugin for Eclipse which, once installed, reveals the abstract syntax tree and connects tree navigation to text navigation. I suspect that we have many of the raw ingredients we need to move forward. What's missing is a coherent UI metaphor that combines the benefits of structure editing and text editing.

Maybe that's an unattainable goal. But I hope people will keep trying to achieve it nonetheless. It would be great if those who have been there and done that could not merely describe the pitfalls, but also show them to us. Why not take a cue from Edwards and use the screencast medium to do that?

The deep structure of code

2005-01-19T10:14:09-05:00

It's been a while since I've written code in Java. But this week I updated my installation of Eclipse, dusted off an old Java program, and saw that ancient artifact in a whole new way. Now, I've never been an IDE (Integrated Development Environment) kind of guy. It's always been emacs for me, and in InfoWorld circles I'm not alone in that preference. Both Chad Dickerson and Maggie Biggs have also made the case for the no-frills approach. At the end of the day, programming still comes down to pushing lines of code around in text files. Until we change that paradigm, I've always thought, code editors built to push lines of code around in text files are going to be hard to beat.

...

At one time or another, every programmer has imagined what it would be like to work directly with the deep structure of code. Some of the best minds in the business are working to make that happen. The legendary Charles Simonyi, who left Microsoft a couple of years ago to pursue his vision of intentional programming, says deep structure is at the core of the toolset his new company, Intentional Software, is building. Sergey Dmitriev shares Simonyi's vision, and his company -- JetBrains, creator of IntelliJ IDEA -- wants to do something similar with its next-generation toolset.

These projects are still under wraps, but another champion of deep structure is working out in the open. Jonathan Edwards, currently a visiting engineer with MIT's Software Design Group, has built a prototype system that he is demonstrating in a screencast at subtextual.org. There are big ideas at work here. In Edwards' prototype, programming, testing, and debugging are just different ways of interacting with a program's tree structure. Edwards' 2004 OOPSLA paper, Example-centric programming, explores one of the benefits of this arrangement: the examples (or "use cases") that drive program design are worked out the context of the living and evolving program.

We've all heard this stuff before. I may yet go to my grave without emacs ever having been pried from my cold dead fingers. But it's worth pondering, now and then, what we could do with tools that didn't think of programs as strings of text. [Full story InfoWorld.com]

In an article for ACM Queue entitled Extensible Programming for the 21st Century (also available here), Greg Wilson skewers the fallacy of "I Want To See My Programs As They Really Are":

Many programmers swear they will never use a system that doesn't let them see their programs "as they really are". However, this position ignores the fact that it takes several hundred thousand lines of device drivers, the operating system, and a graphical interface to turn magnetic spins on a disk into pixels on a screen. If we remove just one of those layers of abstraction, our programs would look like this:
/  *  * r n  |  *  |  T  h  i  s  |  c  l  a
s  s  |  p  r  i  n  t  s  |  <  e  m  >  o  d
d  |  n  u  m  b  e  r  s  <  /  e  m  >  . r
n  |  *  |  S  e  e  |  t  h  e  |  <  a  |  h
r  e  f  =  "  {  @  d  o  c  R  o  o  t  }  /
c  o  p  y  r  i  g  h  t  .  h  t  m  l  "  >
C  o  p  y  r  i  g  h  t  <  /  a  >  . r n
|  *  |  @  a  u  t  h  o  r  |  G  r  e  g  |
W  i  l  s  o  n r n  |  *  |  @  v  e  r  s
i  o  n  |  1  .  2 r n  |  *  / r n r n
p  u  b  l  i  c  |  c  l  a  s  s  |  O  d  d
s  |  { r n  |  |  p  u  b  l  i  c  |  s  t
a  t  i  c  |  v  o  i  d  |  m  a  i  n  (  S
t  r  i  n  g  [  ]  |  a  r  g  s  )  |  {  |
| r n  |  |  |  |  f  o  r  |  (  i  n  t  |
i  =  0  ;  |  i  <  1  0  ;  |  +  +  i  )  |
{  |  | r n  |  |  |  |  |  |  i  f  |  (  i
|  %  |  2  |  =  =  |  0  )  |  { r n  |  |
|  |  |  |  |  |  S  y  s  t  e  m  .  o  u  t
.  p  r  i  n  t  l  n  (  i  )  ; r n  |  |
|  |  |  |  } r n  |  |  |  |  } r n  |  |
} r n  } r n
Remove another layer of abstraction, and we would have a table of numeric character codes; another, and we would have a stream of bits. It is therefore hypocritical to object to adding another layer (unless the person raising the objection still uses a text-only web browser like Lynx [Lynx]. It also ignores the fact that fewer and fewer documents are stored as byte-per-character ASCII; increasingly, documents use some character encoding scheme to represent Unicode [Spolsky], which editors then interpret on the fly.

Somewhat to Greg's chagrin, his article was recently Slashdotted. In response he has reiterated that while XML might provide a standard representation of programs at one level of abstraction, the goal is not XSLT writ large. Sure, you can use a text editor to wrap angle brackets around programming-language keywords, but as the Slashdotters point out there's no real traction there. What most of the Slashdotters ignored was the idea of tools that work directly with the deep structure of code -- transforming it, animating it, extending it.

I've talked with Greg a few times about this topic, and to be honest my end of the conversation was mainly footdragging and devil's advocacy. Bootstrapping an ecosystem full of tools that work this way does seem like an insurmountable problem. Still, times change. Refactoring IDEs are a relatively new thing, but a generation of programmers who have grown up using them will bring new expectations to the game.

Building stuff on top of stuff

2005-01-18T08:37:28-05:00

at XML 2003

We had a really interesting conversation with Adam Bosworth last week. Adam elaborated on the model of distributed query that I first heard him sketch out in his XML 2003 keynote. The focus then was on the (not-yet-announced) Alchemy -- an idea that will have to play out sooner or later, but whose initial open source implementation at BEA became a mystery when Adam moved over to Google. In our conversation last week, the focus was instead on the notion of a network of "queryers and servers" in which the access paths to data look more like what humans do when they traverse the web than they look like what databases do when they consult their indexes.

Clearly XML is a key enabler of this network, and in this two-minute clip Adam explains how and why RSS, in particular, provides the necessary traction:

It's like Lego. When I was a small kid I used to play with these blocks, and build these very complicated buildings with my sister. And the buildings were all made out of very simple blocks. The fact that they all plugged into each other was what made Lego so amazing. The reason I went to XML in the first place was to try to make it easier to plug things together, but nonetheless the right way to think about XML at this point is that each block has an extraordinarily fine-grained machined shape to each knob that extrudes from it. And to plug another block on top of it you have to get a receptacle for exactly that shape, give or take a little bit.

The cool thing about RSS, as people are discovering, as people like Bloglines are showing, as people like Feedster show when you can do a query and syndicate the query into Bloglines, is that the blocks start to be able to plug into each other because pretty much everyone can extrude something where they understand enough of the shape and how to consume it, and they in turn can extrude that shape, again like the Lego block where you can plug a little knob into the bottom of something and out comes the knob at the top.

So that gives you a way to build stuff on top of stuff, and I think that's really exciting. That's sort of what I've been waiting for, for quite a while, ever since I started working with Jean Paoli and Tim Bray and others on this stuff, was not just a way to exchange data, but for certain standards to emerge that everyone could agree on that would be lingua francas. There's still additional information that might be carried along, but there has to be enough shared grammar for it to work. Whether it's intentional and it's Dave Winer's genius, or whether it's partly luck, or whether it's a combination so it doesn't matter, I think this has turned out to be one very useful lingua franca for plugging things into things.

InfoWorld has a pair of SOA (services-oriented architecture) events coming up in May which I'm helping to plan. Here's one question I'd like to put on the agenda. As a language of contracts, SOA involves complex shapes. But as a Lego system, it's about simple shapes. Where's the sweet spot?

Update: I went back and dug up my initial report on Alchemy, from an earlier Gillmor Gang show. Listening to this 3-minute segment -- in which I try to explain Adam Bosworth's notion of injecting data intelligence into the browser -- reminded me why this is a thread we ought not lose sight of.