disLEXia

Friday, 03. January 2003

Video killed the Radio star.

This weblog has been in bad shape recently. Besides reasons like moving to a new city, moving to my computer (1GHz/1GB Powerbook), ruining my new computer (our 19 Month old boy poured a glass of juice into it ruining the main board permanently), moving back to my old machine and moving our whole network (C0RE-NET@RIPE) to another location I had big trouble with the weblog software I used: Radio Userland (http://radio.userland.com/).

For many people Radio Userland might be exactly the right thing and this software is certainly full of good ideas ... but the way I used it was mainly a slow, buggy monster. Some things I want from weblog software:

• I hate waiting for software. With normal user interaction there should be no noticeable delays. Web-based GUIs are slow by design.
  
• A Webbrowser is not a texteditor. (But using (X)Emacs as a webbrowser helps)
  
• The software should scale nicely. "disLEXia" today consists of nearly 4100 items and the software should handle further grow with ease.
  
• I use "disLEXia" to collect old computer crime cases. So posting to the past is essential.
  
• 'Upstreaming', the transfer to the public server must happen in a secure manner. No clear text passwords. I to tend spend significant time in 'hostile networks', and clear text passwords are not only a security risk but everybody would make fun of me.
  
• Every posting/item should have an own page.
  
• There should be categories. (Radio has this feature)
  
• The Software should generate static HTML. (Radio does so)
  
• Data should be stored in some standard compliant way so it can be read by other software - data usually survives longer than programs.
  
• It should work in a way a Unix weeny can enjoy. Counting items should work like this:

% find db/weblog/items/ -name '*.pickle' | wc -l
4079

Radio does not meet all of this requirements - that not it's fault. But it also has some real defects:

• It crashes regularly, at least on MacOS X. While doing so, it takes data with it. The crashes occur in the underlying database/interpreter engine not in the High-Level UserTalk code Radio is written in. One time it took 200 postings with it, it seems Userland is unwilling or unable to get theis Software into a stable condition.
  
• It generates invalid HTML and XML. And there is no single place to fix it.
  
• It is slow. It chews all processor time it can get even when idle. This is a big problem when working on battery power. It takes day to re-render the whole weblog.

Then there is a GUI coded using wxPython which displays the news in a way optimized for speed reading and lets me post items to the (then Radio based) weblog via the metaWeblog XML-RPC-API.

I hacked Radio to generate a single page for each posting. Permalinks pointed to this pages. It turned out rather difficult to get the RSS generated by radio to also point to this pages. Also my Radio extension noted the date I intended the post to move to. Because of Radios implementation Details the Items couldn't be moved to their intended date at once. They where rendered on the actual page. I had to start a script which completely reorganized all posts to move new postings to their intended date. Programming in UserTalk is kind of fun. The persistence features make a lot of stuff very comfortable.

After Radio had lost two days of work in just another crash, I decided to replace it by my own code. Adding a Web server and XML-RPC server to my RSS fetching daemon seems like a natural choice. The whole thing is based on the elegant Medusa toolkit and does all I/O in non-blocking, async fashion (I love I/O multiplexing and financed a considerable part of my Ph.D. time by writing multiplexing network servers - poll()/kqueues are my dearest APIs). It generates pages with the cheetah template engine and is considered fast enough by me. A page with a single item is generated, rendered and served via http in just under 0.01 seconds on my 333 MHz G3 Powerbook. All data is transferred via rsync-over-ssh to my public server. Rendering all weblog pages and transferring them via a 144kbps DSL line to my server takes under 30 minutes. It also emulates a subset of the Ampheta Desk, Headline Viewer and Radio Userland URL space so all this nifty auto-subscribe buttons work.

While being mostly finished there are still some missing bits and pieces. Probably the biggest pice is the rendering of postings. It is nonexistent which means text is just thrown into the HTML template. I still have to add something which adds markup where needed without messing with markup already in the postings.

By the way, the Software is called Tv.
15:50 | #

disLEXia, a research project by Maximillian Dornseif