A sandwich. A good one.

White Paper Bag
What’s inside?

While in Burlington, Vermont, I worked at the excellent Office Squared downtown coworking space. Some days I had to go out for lunch. That one burrito place on Church Street was unworthy of mention. My next try was Top of the Block Sandwich Shoppe. Here I found my usual.

My first time there, I was overwhelmed by the endless options on the sandwich menu. I didn’t want to spend half my lunch break studying a food list. So I simply asked for a sandwich. Any sandwich would do, I said, as long as it’s good. No, I don’t have any food allergies or restrictions. Just make it a sandwich, not a wrap. Surprise me.

This intrigued the young ladies behind the counter. The lucky one who got to make my sandwich was thrilled by the possibilities. She could make whatever she fancied and I would probably love it. The others kept an eye on the project, jealous of her creative opportunity.

I took my sandwich back to the office and I loved it. The bread, the special mayo, the juicy meat and unconventional mix of veggies. This was a unique sandwich made by a person whose curiosity and passion had been aroused by a simple, open-minded request: make me a sandwich. A good one.

Each week I went back once or twice for a new sandwich. Each visit brought smiles to the faces behind the counter. Each sandwich was better than my memory of the last.

By the third visit I didn’t have to speak my order. My request was so memorable that I was an instant regular. I am the guy who wants a random sandwich. Surprise me.

My surprise sandwich was the most consistently pleasurable lunch outside of my home. For the same price everyone else paid for their made-to-order sandwich I got joy, suspense, and possibly the best sandwich made that day.

When ordering a sandwich what I really want is two slices of bread surrounding something good. Turkey, peppers, mustard, none of the above, whatever. Maybe not every sandwich maker has the passion or the sense of humor to do it right like Top of the Block. Maybe I’d be unhappy with the “surprise item” from most other places. I’m happy I found these professionals and took that chance.

How to end fear of the dark

Fear of the darkMy 2-year-old daughter suddenly became afraid of the dark. “I’m scared of the dark!” became her screaming mantra at bedtime and her objection to entering any room without an electric light on–even when there was still daylight. This is how I helped to ease her back into a painless bedtime routine. Continue reading How to end fear of the dark

svn merge -c

How many times have you frantically googled “svn merge revert” to remember how to undo a recent commit? If you answered “too many” then you were probably confused by the syntax of svn merge -rARG1:ARG2. After this post you’ll never have that problem again because you can easily memorize svn merge -c-ARG instead.

# to undo changeset 1234 in the working copy:
svn merge -c-1234 files

The c is for “change” and the - means “in reverse”. When you read that command to yourself, say “svn merge change in reverse 1234”. That is sensible enough that you can remember it forever, even in an emergency. Right?

To apply several changes at once, use several -c- options:

# to undo two changesets at once:
svn merge -c-1234 -c-1220 files

To re-apply the changes in your working copy for debugging, use -c:

# to re-apply the changes you undid:
svn merge -c1220 -c1234 files

May this knowledge shave precious seconds off your next svn emergency!

Programming is not Algebra

A friend discovered a small interest in learning to program computers. She tried an online course but she didn’t get far before losing interest. She may have had other reasons for giving it up but by watching her and asking questions I did locate one significant hurdle: she was trying to read code through the lens of algebra.

I have seen this before. A student encounters a word, symbol, or pattern they recognize from some other context. Believing they have sufficient information to understand it in the new context, they push ahead. Soon their mental model fills with contradictions and the entire subject becomes intractable. This state of affairs causes people to quit their studies.

I’ve been there. Fortunately I learned the solution:

  1. Find what word or symbol was misunderstood.
  2. Learn the correct meaning.
  3. Re-study the material from the first occurrence.

The hardest part is completing the first step. This is especially true of students who already know a great deal and who are confident in their knowledge. Preconceptions hide themselves well but they surrender easily once found. Most people will be eager to complete the remaining steps after they have identified a real misunderstanding.

In this article I will expose some common preconceptions which can cause students of programming to quit. Teachers of beginner programming classes and writers of books would do well to treat these handicaps before setting students to code.
Continue reading Programming is not Algebra

Atomically update serialized PHP arrays in MySQL

Okay, okay, it’s hard to find a use case for this when it’s so obvious that the correct way to handle one-to-many is with JOIN. But if you’re already committed to your schema and you decide you need to append serialized PHP data to a row atomically, you can cons serialized values with this query:

  serialized = "i:1;"
    serialized = CONCAT(

After you have performed this three times with the serialized values 1, 2, and 3, the row contains this:


After unserializing, deconstruct it with this function:

function decons($list) {
    $res = array();
    while ( $list != array() ) {
        if ( $list[0] === 'cons' ) {
            array_unshift( $res, $list[1] );
            $list = $list[2];
        } else {
            array_unshift( $res, $list );
    return $res;

The result:

array(1, 2, 3)

I haven’t actually used it (probably never will) but you are welcome to try this at home!

Proving that this is stupid is left as an exercise for the reader.

Erlang is a hoarder

One day you set aside a shoebox to store newspaper clippings. Suddenly you are trapped under an avalanche of whole newspapers and wondering how long your body will lie there before anyone misses you.

That is what kept happening to my Erlang apps. They would store obsolete binary data in memory until memory filled up. Then they would go into swap and become unresponsive and unrecoverable. Eventually somebody would notice the smell and restart the server.

The problem seems to be related to Erlang’s memory management optimizations. Sometimes an optimization becomes pathological. If you store a piece of binary data for a while (a newspaper clipping) Erlang “optimizes” by remembering the whole binary (the newspaper). When you remove all references to that data (toss the clipping) Erlang sometimes fails to purge the data (lets the newspapers pile up everywhere). If nobody shows up to collect the garbage, Erlang dies an embarrassing death.

The first step to recovery is to monitor the app’s memory footprint and log in every so often to sweep out the detritus. It can be tricky to find the PIDs that need attention and tragic if you arrive too late. The permanent solution is to build periodic garbage collection into the app. It’s not hard to do. The only hazard is doing it too often since it incurs some CPU overhead.

Each time I have found an app doing this, I’ve had to locate the offending module and install explicit garbage collection. If there is a periodic event, such as a timeout that happens every second, I’ll use it to call something like this:

gc(Tick) ->
    case Tick rem 60 of
        0 -> erlang:garbage_collect(self());
        _ -> ok

Today I installed this simple code and here is the result:

Memory footprint reduced drastically
Memory footprint reduced drastically

CPU utilization raised slightly
CPU utilization raised slightly

For the cost of 5% of one CPU core I stopped the cycle of swap and restart. I would like to learn why my binaries are not being garbage collected automatically. The processes involved queue the binaries in lists for a short time, then send them to socket loops which dispose of them via gen_tcp:send/2. Setting fullsweep_after to 0 had no effect. I’ll be interested in any theories. However, I’m not looking for a new solution since mine is satisfactory. I hope other Erlang hackers find it useful.

2011 in review

The WordPress.com stats helper monkeys prepared a 2011 annual report for this blog.

Here’s an excerpt:

The concert hall at the Syndey Opera House holds 2,700 people. This blog was viewed about 46,000 times in 2011. If it were a concert at Sydney Opera House, it would take about 17 sold-out performances for that many people to see it.

Click here to see the complete report.