disLEXia

Friday, 03. January 2003

Video killed the Radio star.

This weblog has been in bad shape recently. Besides reasons like moving to a new city, moving to my computer (1GHz/1GB Powerbook), ruining my new computer (our 19 Month old boy poured a glass of juice into it ruining the main board permanently), moving back to my old machine and moving our whole network (C0RE-NET@RIPE) to another location I had big trouble with the weblog software I used: Radio Userland (http://radio.userland.com/).

For many people Radio Userland might be exactly the right thing and this software is certainly full of good ideas ... but the way I used it was mainly a slow, buggy monster. Some things I want from weblog software:

• I hate waiting for software. With normal user interaction there should be no noticeable delays. Web-based GUIs are slow by design.
  
• A Webbrowser is not a texteditor. (But using (X)Emacs as a webbrowser helps)
  
• The software should scale nicely. "disLEXia" today consists of nearly 4100 items and the software should handle further grow with ease.
  
• I use "disLEXia" to collect old computer crime cases. So posting to the past is essential.
  
• 'Upstreaming', the transfer to the public server must happen in a secure manner. No clear text passwords. I to tend spend significant time in 'hostile networks', and clear text passwords are not only a security risk but everybody would make fun of me.
  
• Every posting/item should have an own page.
  
• There should be categories. (Radio has this feature)
  
• The Software should generate static HTML. (Radio does so)
  
• Data should be stored in some standard compliant way so it can be read by other software - data usually survives longer than programs.
  
• It should work in a way a Unix weeny can enjoy. Counting items should work like this:

% find db/weblog/items/ -name '*.pickle' | wc -l
4079

Radio does not meet all of this requirements - that not it's fault. But it also has some real defects:

• It crashes regularly, at least on MacOS X. While doing so, it takes data with it. The crashes occur in the underlying database/interpreter engine not in the High-Level UserTalk code Radio is written in. One time it took 200 postings with it, it seems Userland is unwilling or unable to get theis Software into a stable condition.
  
• It generates invalid HTML and XML. And there is no single place to fix it.
  
• It is slow. It chews all processor time it can get even when idle. This is a big problem when working on battery power. It takes day to re-render the whole weblog.

Then there is a GUI coded using wxPython which displays the news in a way optimized for speed reading and lets me post items to the (then Radio based) weblog via the metaWeblog XML-RPC-API.

I hacked Radio to generate a single page for each posting. Permalinks pointed to this pages. It turned out rather difficult to get the RSS generated by radio to also point to this pages. Also my Radio extension noted the date I intended the post to move to. Because of Radios implementation Details the Items couldn't be moved to their intended date at once. They where rendered on the actual page. I had to start a script which completely reorganized all posts to move new postings to their intended date. Programming in UserTalk is kind of fun. The persistence features make a lot of stuff very comfortable.

After Radio had lost two days of work in just another crash, I decided to replace it by my own code. Adding a Web server and XML-RPC server to my RSS fetching daemon seems like a natural choice. The whole thing is based on the elegant Medusa toolkit and does all I/O in non-blocking, async fashion (I love I/O multiplexing and financed a considerable part of my Ph.D. time by writing multiplexing network servers - poll()/kqueues are my dearest APIs). It generates pages with the cheetah template engine and is considered fast enough by me. A page with a single item is generated, rendered and served via http in just under 0.01 seconds on my 333 MHz G3 Powerbook. All data is transferred via rsync-over-ssh to my public server. Rendering all weblog pages and transferring them via a 144kbps DSL line to my server takes under 30 minutes. It also emulates a subset of the Ampheta Desk, Headline Viewer and Radio Userland URL space so all this nifty auto-subscribe buttons work.

While being mostly finished there are still some missing bits and pieces. Probably the biggest pice is the rendering of postings. It is nonexistent which means text is just thrown into the HTML template. I still have to add something which adds markup where needed without messing with markup already in the postings.

By the way, the Software is called Tv.
15:50 | #

Monday, 09. December 2002

Making 8.0.9b1 work

The solution. From the QuickScript window enter the following line.

user.im.prefs.flConnectAtStartup = false

Now from the Radio > Outliner > Preferences menu. Select Instant Messaging Notification. Uncheck all the boxes. Something in the Notification via IM doesn't work. I think it has to do with the jabber code and I haven't tested the AIM code.

Hopefully Jake will make some changes to this pref page so that when the first box is unchecked it will make user.im.prefs.flConnectAtStartup = false. [Surgical Diversions]
17:19 | #

Is Radio 8.0.9b1 really the fix?

Jake says he's got Radio fixed with OS X 10.2.2 but some of us don't think it's quite right.

I get stalled out of the system.startup.startupscript. It seems to err in tcp.im.callbacks.startup, where I'm not sure but it causes problems.

The symptoms are the the webserver doesn't start and the Radio > Web Server menu shows '=system.menus.scripts.serverMenuItem () + " Server"' instead of the usual 'Stop Server'.

Rerunning the system.startup.startupscript seems to run through to completion but the DWHP doesn't open automatically like it used to. Everything else seems to be normal at this point -- but I have my doubts. Hopefully we'll get to the solution. [Surgical Diversions]
17:18 | #

Thursday, 05. December 2002

trying to move this blog to my new computer

I'm breathing a big sigh of relief - I think that I just managed to avoid erasing everything I'd done on this blog in the last 2 months. Well I guess that's something positive to be said for my mistakes in the past - I'm now more cautious and managed to avoid something quite unpleasant.

On the other hand, although I was able to save this blog, I wasn't successful in moving it to my new iMac. And I'm giving up for the time being. I'm a bit annoyed at Radio Userland and its documentation. Moving a blog to a different computer isn't such a strange thing to do. Why can't there be some clear, step-by-step instructions to do this for the OS X version? Suggestions, anyone? [explodedlibrary.info]

The problem is the fact that Radio uses hardcodes paths and volume names to find it's files. I see no reason why this sould be good engeneering.

Basically there are two ways to move Radio to a new machine:
1. ensure that the volume, the folder and all parent folders of your Radio installation are named the same on the new machine.
2. Use a tool to rename the internal paths in Radio. This tool is out there, but I forgot where to find it.
16:20 | #

Friday, 29. November 2002

Two other crashes - one while viewing the EventLog and one while viewing the Homepage. One Crash took data with it.
12:33 | #

Two other crashes - one while viewing the EventLog and one while viewing the Homepage. One Crash took data with it.
11:49 | #

Two other crashes - one while viewing the EventLog and one while viewing the Homepage. One Crash took data with it.
11:08 | #

Thursday, 28. November 2002

Radio Crashed (Out of Memory) Taking two days of programming with it.
23:27 | #

Saturday, 23. November 2002

Radio now crases approximately every 12 hours which is unacceptable but I can bear it.
08:28 | #

Thursday, 21. November 2002

Radio now crashes every 10 minutes.
23:51 | #

Monday, 18. November 2002

Per Posting Pages

Ihave hacked Radio Userland to create page per Item like most other weblog software does. The permalink (#) now links to this page. This pages still have issues, e.g.

tags are not rendered correctly, the date and the categories of the posting is not shown, etc.
06:47 | #

Saturday, 16. November 2002

Radio Userland crashed again taking the translate function, a GUID related function and two postings with it. Great.
20:12 | #

Friday, 15. November 2002

Translating Newsitems

German language content in disLEXia now has a "Translate" link which should take you to a machine translation of the item.
23:57 | #

Monday, 11. November 2002

Yesterday I wrote a very simple suite in Frontier called aggyBaby, that uses the new aggregator XML-RPC interface to build a very bare-bones but super-fast static page of the latest stories from the channels I've subscribed to. I think I'll release the UserTalk code in the hope that someone will convert it to Python and/or Java. [Scripting News]
01:19 | #

Saturday, 09. November 2002

Heads-up, we're making some changes to the XML-RPC interface for UserLand's aggregator. It's definitely not frozen yet. The implementation is out to the Radio-Dev list, but that's not frozen yet either. Still diggin! [Scripting News]
19:25 | #

Wednesday, 06. November 2002

Radio refuses to do upstreaming. gnarf!
09:42 | #

Tuesday, 15. October 2002

Internet Again

10 Minutes ago the "SYNC" LED on our ADSL modem switchd from red to green. So we are on the net again. Theoretically we where before connected via an Apple Airport with Modem but some confusing Interactions between the ISDN-to-analog converter in our PBX and the Airport resulted in a transferrate of ca. 800 b/s and RTT latency of up to 22000 ms (22 seconds!). So basically Internet was unusable.
11:10 | #

Sunday, 06. October 2002

John Robb

John Robb: "Over the next couple of months we plan to focus on making sure Radio and Frontier are rock solid. In most cases it is, but in a minority of situations we need to make improvements." [lawrence's notebook]
23:03 | #

disLEXia, a research project by Maximillian Dornseif