A Quick Note about Planet Awn

Weirdness which I can now attribute to "PHP Library Hell" (similar to DLL hell) had caused the full text of posts for most of the feeds to not be syndicated in the planet feed for several weeks. This is now fixed. (I had to upgrade to Wordpress 2.8 for the actual error to appear and a solution to be presented, it seems.) I apologize for the inconvenience.

Avant Window Navigator (Awn) 0.4 Progress Report: June 2009

I've been a bit quiet about Awn in my blog. This is partially because I've been working on two major areas: libdesktop-agnostic and Vala/GObject Introspection (as explained in my previous post, On Bindings).

Since my work on libdesktop-agnostic is directly related to Awn, I'll address it here (even though it may be boring to most of you). I currently have abstractions for configuration, desktop entries, and virtual file systems (e.g., GIO). The area that needs the most work currently is the configuration support. What will ultimately happen is that there will be a class which lets you have per-instance configuration for a given app(let), regardless of the backend that is being used. What this means is that Awn users won't be shackled to using GConf if they want a stable dock. Plus, if a user wants to switch backends for some bizarre reason, they don't have to recompile Awn to do so. Obviously, they'd have to migrate the settings themselves, but that's the price one pays.

Anyway, on to Awn. Here's a screencast I took (CC-BY-SA 3.0 licenced!) of a development version of Awn in an Ubuntu Jaunty virtual machine (hooray for VirtualBox!):

(Direct link to YouTube video)

The bar colors are kind of ugly, mostly because I was testing some of the color-related code on that VM.

Obviously, the features shown in the video (orientation, panel style, and applet loading indicator) aren't the only new features in 0.4. I'll be showing off more shininess in successive videos.

Moonbeam has covered most of the current status of Awn 0.4, which I will repeat here, briefly, for those of you who are scared of clicking on links:

• Moonbeam is working on an API that allows applets to have text and graphics overlay the applet icon. Think Awn plugin support for applets.
• He is also rewriting Awn System Monitor so that one can monitor multiple things from the dock, among other things.
• Certain panel/task animations still need to be implemented.
• Plugin (not applet) support still needs to be implemented.

I'd like to take a moment to talk about the Awn plugin system, particularly to those who actually write the plugins. The D-Bus plugin API that was in 0.2.x and 0.3.x will be deprecated in the 0.4 series, and removed in the 0.6 series. There will be a new API in the 0.4 series.

With regards to what I'm working on - in addition to libdesktop-agnostic integration, I will most likely be working on implementing the D-Bus plugin API, once Moonbeam is done with the API mentioned above.

On the Awn Extras front, I have the Garbage applet waiting to be added. This is waiting on Vala support being re-activated (which is dependant upon adding GObject Introspection support to Awn), and porting the applet to the 0.4 API. There is also an rTorrent frontend and a social aggregator applet that I have on the backburner (both written in Vala), which may or may not make it into 0.4.0.

And finally, a note for Ubuntu users: we are not making available PPA builds of 0.4 until there are no feature regressions in the rewrite. This release is targeted for the official Karmic Koala repositories, just as 0.3.2 was targeted for the Jaunty Jackalope repositories. We anticipate building packages for Hardy and above.

Remember, the best way to keep track of new developments in Awn/Awn Extras is to subscribe to Planet Awn. Be with us next time for "Autohide and Seek", or "The Incredible Shrinking Dock"!

Rshrtnr: The private URI shortener

Over the past couple of days, I've implemented a private URI shortener service for myself, which I have named "Rshrtnr". The derivation of the name is left as an exercise for the reader.

My main motivation for writing it was a criticism of public URI shortening services that I have been seeing in blogs for a long time: if the service has some downtime or suddenly disappears, all of the links that you have created with it are useless. With my approach, I regain some control of where my shortened links point to, and if the service has downtime and/or disappears, I have more options for restoring it.

The code itself is written in Python. SLOCCount says that the core module runs at around 100 SLOC. Most of my time, however, was taken up by working around problems relating to my webhost's Python installation. The supported Python version is 2.4.x, which is ridiculously old (for reference, Gentoo was the last major Linux distribution to switch from Python 2.4 to 2.5, around July 2008). Additionally, for some reason, if I attempt to change the sys.path variable (i.e., the "include path") to use locally installed modules (I am on a shared host), the entire script breaks with zero logged messages anywhere. It runs fine via the command line, but in FastCGI mode, the strangeness occurs.

The two third-party modules that I used were Paste and mysql-python. I store the URIs and their associated aliases in a simple SQL table, and I use Paste for various WSGI/HTTP-related utilities. I "manually" handle routing via parsing the PATH_INFO environment variable.

There are two ways to specify an alias: either explicitly send a custom one as a query parameter with the URI, or let the app make a random one for you. With the latter behavior, it hashes the URI to generate an eight character "unique" alias. Since there are (in theory) 64^8 possibilities, I don't think I'll run out of aliases any time soon, especially since custom aliases can be anywhere from 1 to 15 characters long.

In my opinion, the most interesting feature is that adding URIs requires one to send an OpenPGP-encoded query string, which needs a public key recognized by the app for the operation to succeed. To write this, I simply parsed the output from sending the OpenPGP message to the gpg binary.

Finally, mod_rewrite magic is used to prettify the shortened URIs. Nothing too exciting about that part.

I had thought about hosting a version of Rshrtnr on Google App Engine, but a key component is missing - OpenPGP support.

If anyone wants me to release it, please comment below. There's currently a bunch of webhost-specific things that I would need to abstract out before I release the code to the general public, and unless someone gives me a very good reason, it will be licensed under the AGPL version 3.

On Bindings

One of the more interesting areas in software development, to me at least, is language bindings. Being able to interface with a library written in one language in another language is kind of satisfying, as it allows me to develop without having to reinvent the wheel. There are two specific projects that I use and work on so that I can enhance the software that I develop: GObject Introspection (G-I) and python-spidermonkey.

GObject Introspection

As a quick overview, the goal of this project is to give C libraries the tools to provide enough metadata about their API so that bindings can be written with minimal effort. Given the time and effort that I have put into maintaining the Awn bindings, it is not very surprising that I would be willing to help out getting this framework working for Awn. My ultimate goal is to eliminate the bindings/python folder in the Awn source tree. It is basically a mixture of a Scheme definition file plus a very bizarrely formatted "override" file for custom definitions, all integrated into autotools to produce a C library that is ready to be dynamically loaded into python via import. To meet this goal, I am contributing to the PyBank project, which is a prototype Python module that interfaces with the GObject Introspection library to read compiled library metadata files (called "typelibs") on the fly so that classes, functions, etc. can be loaded and called at runtime. In addition to myself, a Google Summer of Code student and a Sugar Labs developer are also working on the module, with Johan Dahlin overseeing it all. So far, I've contributed a unit test suite, ported from the gjs project (JavaScript bindings for GLib-based libraries based on the Spidermonkey VM) and working type bindings for various simple types (e.g., int64 and float).

I have also put some coding effort toward G-I integration in Vala. Vala supports G-I by both reading GIR files (the XML serialization of G-I metadata) to produce VAPI files (short for Vala API files), and writing GIR files when producing a library written in Vala (e.g., libdesktop-agnostic). I have contributed mostly what amounts to workarounds in the GIR reading code, with regards to Vala/G-I behavioral inconsistencies. Didier 'Ptitjes' has done much, much more solid work than I have on both fronts, which I greatly appreciate.

python-spidermonkey

This project, as the README states, lets you [execute] arbitrary JavaScript code from Python[, and allows] you to reference arbitrary Python objects and functions in the JavaScript VM. As I've stated in an earlier blog post, I use this in my custom website build system to both validate and pack my JavaScript code, via JSLint and Packer, respectively. Since I published that post almost two years ago, that project was revived twice - once by a Mozilla employee (and co-founder of Humanized, which is quite awesome) named Atul Varma, and the latest incarnation is on github. Since it is based on the original implementation in C, and not the Python-based ctypes version, the Base2 recursion problem does not exist, and so I have happily written modules and scripts which wrap the two JavaScript utilities. Recently, I have made them available in a public project on Launchpad called python-jsutils. I haven't really announced it until now because it currently relies on a change I made to python-spidermonkey which allows one to iterate over a JavaScript array, instead of having to write "unpythonic" code like for x in range(0, len(foo)): #.... While it is in my fork, it has not been merged to the "official" repository.

Website Internals: The JavaScript Tag Module

I need something to get myself blogging again, so I figured that I should write about how my website is written. I'm going to start with one of the parts written in JavaScript: the Tag module. It's comprised of a simple JavaScript object with static methods and properties, no instances. Its main purpose is to create (X)HTML nodes for use in what used to be known as "dynamic" HTML. Incidentally, it's also part of my "old projects" series: I originally created it for an Intel-sponsored research project that I was working on during my time in university. (For more details on the project, please see my curriculum vitae.)

In practice, calls using Tag.create() don't look too horrible. Consider the example of building a minimal HTML5 document:

var html = Tag.create('html', {children: [
    Tag.create('head', {children: Tag.create('title')),
    Tag.create('body')
]});

Or, a data table:

var table = Tag.create('table', {
    attributes: {summary: 'MLS Statistics'},
    classes: ['sortable', 'centered'], // uses sorttable 
    styles: {border: '1px red inset'}, // use CSS style names, not JS ones
    children: [
        Tag.create('thead', {children: Tag.create('tr', {
            // text is automatically converted
            Tag.create('th', {children: 'Team'}),
            Tag.create('th', {children: 'Goals For'}),
            Tag.create('th', {children: 'Goals Against'}) 
        })}),
        Tag.create('tbody', {children: [
            Tag.create('tr', {
        Tag.create('tbody', {children: [
            Tag.create('tr', {
                classes: ['west-coast', 'usa'],
                children: [
                    Tag.create('td', {children: 'Seattle Sounders FC'}),
                    Tag.create('td', {children: '9'}),
                    Tag.create('td', {children: '3'})
                ]
            }),
            // add more teams here...
        ])
    ]
});

I recently modified the function so that it deserializes an object (originally a JSON string) into a DOM node tree. This is particularly useful when you're sending partial HTML documents as JSON strings (which I prefer to sending HTML strings and dealing with that mess). So, the first example would look like this:

var html = Tag.create({
    "name": "html",
    "children": [
        {
            "name": "head",
            "children": {"name": "title"}
        },
        {"name": "body"}
    ]
});

Interestingly enough, I created this without the knowledge of the existence of JSONML or any of its bretheren, although I suspected that JSON-HTML converters already existed.

Three of the functions in the module are basically wrapper functions. Tag.createWithText() creates an HTML element with a text child node, and Tag.createHeader() creates an HTML header element (e.g., <h1>...</h1>) with a text child node. Tag.text() is shorthand for the DOM's document.createTextNode() method.

In the current iteration, nearly all of the event-related code is commented out, as there are far more competent JavaScript libraries out there which deal with cross-browser events. The only event-related function left is Tag.dispatchEvent(), which sends a "synthetic" event for a given HTML element. If I remember correctly, I coded for both the W3C and Microsoft models, but I don't remember testing it on browsers other than IE6/7 and Firefox 2. At some point, I'll probably reintegrate event support to Tag.create() at minimum, using Base2.

The remaining function is a utility function. Tag.inXHTML() determines whether the document in question is in XHTML mode, using a variety of heuristics. I'm sure there's a better way of doing it, but I couldn't find one when I was researching it.

This module has been tested in IE6/7, Firefox 1.5/2/3, Safari 2/3, and Opera 9.5 - although not all in the same time periods. It's licensed under the Apache Licence (version 2.0) and is currently somewhere in my compressed JavaScript file. If there's any interest, I'll put up the non-compressed version somewhere and update this post.

The GNOME Platform, Awn, and the Cloud

Benjamin Otte recently wrote about desktop-web integration in the GNOME desktop. It's kind of interesting that he calls himself "not web enabled", given that he's the main developer of the swfdec library and associated applications. I agree with most of what he wrote, but there are a few comments I would like to make.

Benjamin asks:

Why does dconf (or GConf) not store my settings in the cloud so it uses the same settings on my university login?

As I understand it, this is one of the features of Conduit. There's a bug in Awn regarding config synchronization via Conduit. I'm probably going to look into how that works when I resume work on the config interface for libdesktop-agnostic.

There is an erroneous statement in his post:

We don't even have a http library that knows cookies.

libsoup has had (non-persistent) cookie support since 2.23.1, and persistent support will be in 2.26.0.

And then there's the main point:

[GNOME is] doing a very bad job at integrating the web into the desktop. Apart from epiphany (and the weather applet), what application does GNOME ship that speak http?

I believe there are two main reasons for this: One is GNOME developers are not "web-enabled". [...] The other, related reason is that we don't have the software to do this.

In Awn-land, we have several web-enabled applets:

• arss
• comics
• digg
• lastfm
• meebo
• pandora
• rtm
• weather
• webapplet

In particular, webapplet is a work in progress framework, which will allow what is essentially an applet version of Mozilla Prism. It currently uses WebKit as the backend, although there is also a Gecko-based backend planned. The rest of the applets listed are written in Python. In particular, the digg, meebo, pandora, and rtm applets use the gtkmozembed Python bindings to view the respective websites. Here lies one of the problems of the web-enabling the GNOME platform. This library is acknowledged to be less than ideal. If you look at the source code of the applets, you'll see that there are some ugly hacks in order to make them work properly on Ubuntu systems. Webapplet is slated to replace that ugliness.

One of my side projects is a status aggregator applet. It's supposed to aggregate all of these social networking status feeds and also to sync all of your personal statuses. One of said social networking websites returns complex, site-specific HTML that obviously needs to be sanitized/canonicalized. The easiest (but not necessarily most memory-efficient) method of performing that task is to use the DOM. There is not currently a DOM library for the GNOME platform. There is a libgdom3 project (written in Vala) which I believe is being / will be used by the gnome2-globalmenu project, but it is unfinished. There is also an old, unmaintained library called gdome2 based on libxml2. I'm not even sure if anyone actually uses that library anymore (its last release was in 2003). I avoided using gtkmozembed and friends based on the experiences described above. I settled on a promising feature request for WebKit: a GObject/C DOM binding. (As an aside, the WebKit bug linked is a fascinating case study on several levels: conflicting coding standards, conflicting developer personalities, and some interesting coding/reviewing.) It's very nice - I can manipulate document fragments as if I were using JavaScript in a web page, among other things. I eagerly await that feature being committed to WebKit trunk.It is an important stepping stone when it comes to working with the web.

Another side project that I'm currently working on is a developer dashboard applet. It's kind of like the previous applet, except as applied to software projects. I was originally going to write it in Vala, which meant that I would have to write an interface to (at least) the Launchpad API, which meant implementing at least three specifications in Vala: OAuth, URI templates, and WADL. I finished the first two (I haven't yet decided whether to release my URI templates implementation as a separate library - the implementation plus the test app is 451 source lines of code), and WADL is a very complex specification. So, I decided to postpone working on the WADL library and instead am currently working on a prototype applet using Python and launchpadlib. Implementations for those three specifications and many others (including AtomPub) should be included in the GNOME platform if it wants to be web-enabled.

On Webhooks, or The Push Revolution

Mike Rooney (Awn Extras developer, among other things) posted an article on this "webhooks" idea. As I understand it, it's essentially a customizable, web-based version of the "Subscribe to future XYZ via email" features (e.g., blog comments) that are currently around. The key phrase in this sort of thinking is "push technology". Mike asks:

So will webhooks replace the current paradigm that I'm using here, or complement it?

I believe that it will complement the current paradigm. We need to have a transitional period (à la the old, rigid deadline for the US digital television transition) between the current "polling" techniques and the "new and shiny" (and arguably bandwidth-saving) push techniques. Take, for example, Twitter [1]. Let's assume that they didn't shut down their XMPP service, and they built upon it an (XML-based) API so that a client (for the purposes of this thought experiment, let's say my currently-in-vaporware Status Aggregator Awn applet) could connect to a specific JID (AKA user name + domain + "resource", or specific client) and listen for any new tweets, responses to my tweets, etc., replacing those messy timeout callbacks with messy async socket callbacks. The main benefit that I see is a savings in bandwidth for both the consumer and the producer. It avoids sending network requests every X minutes, which would add up, given the number of services/feeds that a user subscribes to (including mail). This actually leads me to my answer to Mike's other question:

Are webhooks the next step of this evolution, or something else entirely?

As evidenced by my thought experiment, I'd like to see XMPP as the next step, or at minimum, the step after webhooks. While I love HTTP, and am a big fan of the whole REST concept, it's hard for me to see it used as a facilitator for pushing data, as opposed to pulling it. In fact, given the way that ETags and the like are designed, HTTP is inherently a pull technology. The Comet model feels like a big kludge to me for that reason. XMPP, on the other hand, is designed to be a push technology, and its supporters are actively marketing it as such. It's also scalable, as servers like ejabberd and services like Google Talk can attest. I suppose the bottleneck here is a catch-22: you need both services and apps to buy into this particular implementation. Maybe when I finish one of the myriad projects I have going at once, I'll take a crack at adding a push-based web service client. Ideally, for configuration, all a user would have to do is set their app-specific JID (e.g. foo@example.com/bar_app) in both the web app and the client app, and it would "just work" (well, you would also have to set the JID password somehow as well, but that's beside the point).

---

Update (2009/02/18): It seems that I was subconsciously channelling a presentation on XMPP PubSub that I read over six months ago.

---

[1]	I'm focusing on the one I actually use. Yes, I should be using identi.ca, since I am a supporter of free and open services/protocols. It even has the hallowed XMPP interface to microblogging. One of these days, I'll do what all the cool kids™ are doing and post to both. It'll probably happen when I (continue) work on the vaporware [2] mentioned above.
[2]	It's vaporware until I push the source code onto a public server.