Sometimes RSS isn’t fast enough. We’ve been experimenting with faster blog subscription delivery using Jabber to push the messages. When you use Jabber to subscribe to blogs you get the news as soon as it is published. Now almost every post and comment on WordPress.com blogs is published this way and you can subscribe to these streams using almost any Jabber client. Messages are delivered typically within one second of publication. You can also publish to your blogs by typing instant messages. Soon comments will be appearing in Jabber chat rooms.

Let me break it down. Jabber (XMPP) is an instant messaging protocol. There are dozens of free clients (programs) that can connect to Jabber services, thousands of Jabber servers, and millions of daily Jabber users around the world. Even some phones can connect to it, including the iPhone with an appropriate app. So you can probably use Jabber. The primary exception is people behind company or government firewalls that block XMPP ports, but we’ll have a web-based solution for them soon.

Before I give you a link I have to tell you that this service is experimental. If you use it, you are a tester so please wear your white lab coat. A few dozen people have been using it for several months with very few hiccups, but hiccups are still to be expected. Even so, most of us at Automattic rely on it daily to surface and accelerate the discussions on our private blogs. Finally I must tell you that we have not yet worked out all the business angles, so the feature set and limitations may change to accommodate our inevitable need to feed the monkey.

Now I give you im.wordpress.com, which I demonstrated at the CrunchUp last week. Every WordPress.com account is automatically linked to a Jabber account on im.wordpress.com. We have compiled instructions for setting up some popular Jabber clients.

If you want my personal recommendation for a Jabber client, my choice for Mac OS X is Adium. If you already use iChat, just stick with iChat. Some of my coworkers who run Windows have chosen Pidgin. I also sometimes use Psi, which is available for Mac, Windows, and Linux.

Here is some info for people familiar with XMPP. This service is based on XEP-0060 (Publish-Subscribe) acting as a front-end for WordPress blogs. It started as a simple firehose for our commercial partners and grew from there. People subscribing with Jabber clients don’t need Pubsub. They send simple commands to a chat bot and their items are delivered as XHTML-IM from the blog’s URL. The bot speaks XEP-0060 on their behalf. If you can speak XEP-0060, you can connect to pubsub.im.wordpress.com and subscribe to nodes. The nodes for this blog are /blogs/andy.wordpress.com/ for posts, /blogs/andy.wordpress.com/comments/ for all comments, and /blogs/andy.wordpress.com/2009/07/16/real-time-wordpress-com-subscription/ for comments on this post. Node discovery and item discovery and retrieval are not implemented. Reasonable subscription and traffic limits will be imposed. If you are looking for a complete feed of all our blogs and comments, try the firehose.

At first we thought it was a good idea to use iframes to display reports in the Stats plugin. We’ve seen since then a lot of problems with browsers and cookies. To help resolve these issues, and in anticipation of future features, I am updating the plugin and the WordPress.com stats reporting system to remove the iframes. I just posted 1.5 beta 1. If you host your own WordPress 2.7+ blog and you use the Stats plugin, why not contribute to its development by installing this testing version? Anyone can download the beta but I don’t recommend it unless you are able to cope with potentially unstable software.

• What are the risks of using this beta?
  You won’t lose any stats. If something goes horribly wrong it’s probably a bad download; just reinstall the latest version of Stats.
• How does it work?
  The plugin connects to WordPress.com to get the stats reports when you request them. It uses the API key to authenticate.
• Aside from fixing cookie problems, how is this better?
  Now it’s possible for anyone who can publish posts on your blog to see blog stats. They don’t have to be logged into a WordPress.com account. They only need the publish_posts capability (Author role) to view stats reports.
• Where did the dropdown blog switcher go?
  Because the plugin uses a single API key to authenticate, the service doesn’t know whether the visitor is the owner of that key or some other user. So it doesn’t make much sense to show the list of blogs belonging to the API key owner. You can still use the switcher if you view your stats on any WordPress.com dashboard.
• Where did the Stats Access panel go?
  This is also related to single API key authentication. Maybe in future we will bring administrative access back to the plugin. But until then, we have left the Stats Access panel intact on WordPress.com dashboards. You might want to bookmark dashboard.wordpress.com if you need these features on a regular basis.
• Will this be a required upgrade?
  You mean will older version of stats be broken? Not by 1.5. Later versions may break compatibility but for now you can keep using earlier versions of Stats if you like.
• What if I install this and still see iframes?
  This happens because your server is unable to connect to WordPress.com. I set it up to use SSL (https) in the hopes that most hosts support this. If yours does not work, I’d like to hear from you and do some testing on your host.

We usually do things sequentially in PHP. Any point where we can get two things done at the same time is an opportunity to reduce the total execution time. It is generally safe to execute processes in parallel when no process has a side effect that may impact any other process. Most MySQL SELECT statements fall into this class. Under certain conditions it might help to run these queries in parallel.

So I got to thinking: does PHP provide a function to issue a query and return immediately, instead of waiting while the database churns? Such a function would let me spread a batch of queries across several servers and then collect the results. It should not involve forking or making OS calls. Maybe mysql_unbuffered_query is that function.

Assuming it is, here is the basis for a parallel query system in PHP. It is obviously incomplete. I may complete it and try it out when I need to query a dataset partitioned across separate MySQL instances. The function connect_reserved($query) returns a locked MySQL link identifier, opening new connections as needed. The function release($query) removes the lock, returning the link to the pool of available database connections.

function multi_query($queries);
    foreach ( $queries as $i => $query ) {
        $link = connect_reserved($query);
        $res[$i] = mysql_unbuffered_query($query, $link);
        $ret[$i] = array();
    }

    do {
        foreach ( $res as $i => $r ) {
            if ( $row = mysql_fetch_row($r) ) {
                $ret[$i][] = $row;
            } else {
                release($queries[$i]);
                unset($res[$i]);
            }
        }
    } while ( count($res) );

    return $ret;
}

I have not used mysql_unbuffered_query. For best results, it should return as soon as the server determines that the query is valid. I have assumed that it can return while the database is still looking for records. (If it can not, this whole idea should be forgotten.) This oversimplified diagram helps illustrate how the queries run in parallel. The green line shows the beneficial overlap of query processing time.

Running PHP code from within Erlang is easy: os:cmd("php -r 'echo \"Hello, World!\";'"). This is fine when you need to run simple commands. When you demand more from PHP, this approach becomes awkward, wasteful, and eventually unusable. If your Erlang-to-PHP calls require large PHP applications, open connections to databases, or somehow incur significant initialization overhead, you should maintain pool of reusable PHP processes.

My first complete application for Erlang/OTP is php_app. It manages a pool of persistent PHP processes and provides a simple API to evaluate PHP code. I designed php_app to be robust and easy to use. It’s so easy, in fact, that I now use it to debug WordPress functions from within Erlang. Here is a sample session using start/0 and eval/1:

$ erl
Eshell V5.6.4  (abort with ^G)
1> php:start().
ok
2> php:eval("echo 'Hello, World!';
2>           trigger_error('Uh-oh!');
2>           return array(true, true);").
{ok,<<"Hello, World!">>,
    [{0,true},{1,true}],
    <<"Uh-oh!">>,continue}
3> 

In the resulting tuple we have the output, the return value, and the last error. The atom continue indicates that the PHP process is eligible for reuse, determined by its size in memory after evaluating my code. In the next example I’ll reserve a PHP process to demonstrate persistence and what happens when we hit the memory limit.

3> Ref = php:reserve().
#Ref<0.0.0.52>
4> php:eval("$a = array_fill(0, 200000, rand());
4>           return count($a);", Ref).
{ok,<<>>,200000,<<>>,continue}
5> php:eval("$a = array_merge($a, array_fill(0, 200000, rand()));
5>           return count($a);", Ref).
{ok,<<>>,400000,<<>>,break}
6> php:eval("return count($a);", Ref).
{ok,<<>>,0,<<"Undefined variable:  a">>,continue}
7> php:release(Ref).
ok
8>

The function reserve/0 removes a PHP process from the pool and returns a key that is used in eval/2. Without a key, we can’t be sure that the same PHP process will evaluate our next string of code. Notice the correct return value of 400000 and the atom break which indicates that the PHP process has been restarted because it exceeded the memory usage limit. Our Ref now points to a fresh PHP process. The reservation remains valid.

There are a few other return tuples: one for timeouts, one for parse errors, and one for exits. That last one includes fatal errors. They can’t be trapped. You’ll just have to refer to your error logs. (You do write code with a terminal tailing all your error logs, don’t you?) Here are some more bullet points to keep in mind:

• Never define a PHP function without first testing function_exists because you will get a fatal error every time. This is by design.
• Be mindful of escaping quotes and control characters. User input is the enemy. Test.
• This app was written by an Erlang novice. Do not underestimate its potential destructive power.
• Even so, it’s in use on a production system that makes lots of PHP calls.
• The php module has EDoc for all of the API functions.  The HTML version is included for completeness.
• If you modify the PHPLOOP, you should restart the PHP processes. Try php:restart_all().

The code is here. All you have to do is compile it. I like to make:all(). The configuration is in php.app.

If you would like to contribute changes to the code or documentation, I will be happy to hear from you. My OTP stuff could benefit from a more experienced set of hands.

A current project of mine involves the Jabber extension PubSub (XEP-0060) and I chose ejabberd to power it. Ejabberd, the software that powers jabber.org, is written in Erlang. I am having a fantastic adventure learning Erlang. The beginner mind is where I feel most comfortable. But that’s another post.

Recently I wanted to make part of a nodetree virtual. A virtual nodetree is one that doesn’t store any nodes. Thus you can have infinitely many “virtual” nodes with zero records in the pubsub_node table. For any large set of indentically configured, transient nodes, this is an ideal setup.

Ejabberd only supports one nodetree plugin per virtual host, so I created a nodetree plugin that lets me use as many different nodetree plugins as I like. A simple pattern match determines which nodetree plugin to call and calls it. Here is the pattern matching function:

nodetree(["dogs",_|_]) ->
	nodetree_dogs;
nodetree(["cats",_|_]) ->
	nodetree_cats;
nodetree(_) ->
	nodetree_default.

And this is how the rest of the plugin looks:

create_node(Key, Node, Type, Owner, Options) ->
    Nodetree = nodetree(Node),
    Nodetree:create_node(Key, Node, Type, Owner, Options).

Now I can have as many nodetree plugins as I want on a single virtual host. As a bonus, I don’t have to touch mod_pubsub when I add a new nodetree plugin. All I do is reload the nodetree_hybrid module and mod_pubsub never misses a beat. Erlang, you’re my hero.

Self-hosted WordPress and WordPress MU administrators: if you are using the memcached object cache (a prerequisite for batcache), upgrade it before upgrading WordPress. There is a bug that keeps the old db_version in the options cache, preventing WordPress from remembering that it has been upgraded, and this causes it to ask you to upgrade again. In a pinch you can resolve the problem by restarting the memcached daemon.

The memcached object cache can’t be automatically upgraded because it’s not a normal plugin. Make sure to use the right version: 1.0 (sockets) or 2.0 (PECL). Only one line was changed, so you might prefer to update by hand: 1.0, 2.0.

If you aren’t sure whether you are using memcached, look for a file named object-cache.php in wp-content. If that file exists, look inside to see if the plugin name is “Memcached”.

Inspired by the new and sublimely useful WP Roadmap, here is a high-level diagram of WordPress in three sizes: 512, 1024, 2048. Made with OmniGraffle Professional.

Because much of my work happens in IRC, I like to stay attached to the Freenode IRC server all the time. This was easy when I kept a desktop computer always on. But now that I use my laptop exclusively it’s impossible: every time I close the lid my IRC connection closes, so I can’t read what was said and nobody can send me a message.

So I went looking for something that could keep me online even while my laptop sleeps. I found dircproxy, a Detachable IRC Proxy that runs on Linux. It was easy to install from Aptitude in Debian but their version had one very annoying bug, so I compiled the current dircproxy source code and installed the new binary by hand. Now everything is sweet.

When I want to connect to Freenode, I actually connect to dircproxy, which in turn connects to Freenode. When I quit, dircproxy keeps my IRC nickname online and logs everything that happens while I’m away. When I return, dircproxy plays back the logs so I know who said what when, including private messages. It’s like I was never gone.

Aside from solving the basic problem of intermittent connectivity, dircproxy has other benefits. You can make it keep complete logs apart from the ones it uses internally. You can specify different nicknames for when you detach. You can customize messages to send when you detach and reattach, as well as auto-replies for when you are away. It has a long list of options I don’t understand, but they look good.

I was discouraged when I found a lot of dead links, but the project has been uploaded to Google Code and it’s getting some attention again. The Freenode dircproxy channel has 15 members. I hope they can attract more developers to the project. They also need some new links to get them past the old, dead ones. Blog it if you try it.

ENCODED=$(echo -n "value to encode" | \
perl -pe's/([^-_.~A-Za-z0-9])/sprintf("%%%02X", ord($1))/seg');
echo $ENCODED

This command line percent-encodes strings, even binary data. You can use redirection. I used `echo -n` because I didn’t want a newline (%0A) tagged onto the end. I’ll be using this in a bash script to send HTTP POST data via curl.

My result is based on Perl URL Encode & Decode String, modified to be run on the command line and updated to skip the four non-alphanumeric unreserved characters specified in RFC 3986 Section 2.3.

I don’t know if this depends on Perl multibyte settings or anything else. I might have just been lucky. Perl isn’t my thing.

Heh, the Lenny installer lists the keymap “US american” instead of “US English”… is that a joke on the Miss Teen USA flop?