Clustering? What Clustering?

Matt Dillon, of well deserved FreeBSD and DragonFlyBSD fame, has floated a proposal for a new filesystem. Some have called it a cluster filesystem, which seems a bit odd to me since … well, there doesn’t seem to be any actual cluster functionality involved. There’s some stuff about replication, but that seems more oriented toward slow links and disconnected nodes. There’s nothing about issues like coherency and latency reduction and access control across systems and split-brain scenarios and scaling across storage systems … i.e. the kinds of issues that require the most design effort for an actual cluster filesystem. So maybe it’s a “cluster filesystem” not in the sense of providing cluster functionality itself but instead by being more compatible with other cluster software? That would mean failing over better or faster than other filesystems, or perhaps having some extra hooks to exchange cluster-relevant events, but none of those features seem to be there either.

Coda did disconnected clients two decades ago, and its descendants are still going strong. Many projects and products out there do replication, often including extra functionality such as WAN optimization and CDP as well. Where’s the novelty? How is Yet Another Local Filesystem, with not even as robust a feature set as ZFS but with basic replication tacked on, the best use of anyone’s time and energy? If you’re going to all that trouble, Matt, do something more ambitious. I know you have the talent for it.

Coding Quiz

Here are two little gems that I found recently, in a very popular piece of open-source software. See if you can figure out what’s wrong with each before reading on.

    /* d might support PROTO_XXX, PROTO_YYY, both, or neither */
    if (d->proto & PROTO_XXX) {
    if (d->proto & PROTO_YYY) {
    while (!got_reply && !time_expired) {

Note that I very explicitly said at the top that d might support neither protocol. What happens then? We sleep waiting for a reply that could never possibly come to us because we never sent out either kind of request. I thought this was pretty bad, until I saw the next snippet right next door.

    /* Their label name, not mine. */
        retries = 2;
        if (try_to_do_something() == OK) {
            return SUCCESS;
        if (--retries) {
            goto try_try_again;
        return FAILURE;

Will this ever actually return FAILURE? No. Coding the functional equivalent of “while (2 – 1)” is bad enough, but doing it with a goto is just adding insult to injury.

The outcome of these two bugs is that something which should have failed almost immediately and moved on got stuck in an endless loop instead. The only excuse I can make for its author(s) is that it’s relatively unlikely that people will want something to fail, or configure something in a way that would fail. In this case, though, the goal was to use a single configuration in two places, even though it would only work “normally” in one place and fall back to an alternate method in the other, because maintaining two separate configurations would have been worse than having a slight delay in one.


Ingo Molnar has published a Linux patch that introduces “threadlets” – functions that can run within the context of a caller as a simple function call, unless/until they need to block, at which point they are split off into their own threads. Ingo refers to this, not inaccurately, as “on demand parallelism” and it’s pretty cool. I actually know of one other system that did basically the same thing several years ago, except that it was for kernel interrupt handlers, and it was cool then too. I do worry, though, that people will mistakenly think of threadlets as a complete solution to the problem of balancing function-call convenience with separate-thread scalability, when in reality they are more of a complement to existing mechanisms than a replacement. Specifically, threadlets are best when the code within a threadlet meets two criteria:

  • It might block.
  • It probably won’t.

If there is no possibility at all that the code will block, threadlets don’t buy you anything compared to a simple function call. On the other hand, if the code probably will block then threadlets are no better than regular threads. In fact they are regular threads, except without many of the primitives that would be available to manage normal threads. Spawning a threadlet per connection in a network server, for example, would probably be worse than just spawning off a regular thread per connection.

Fortunately, this “might block but probably won’t” space in between is very large. A lot of code doesn’t know whether it will block, because it’s making system calls that hide that information, and hiding it is sort of the whole point of a multi-process operating system. An interface that made blocking more explicit or visible from the user-space code’s point of view would also involve many more kernel/user boundary crossings, which would significantly degrade performance and impede debugging. The result is that an application often has to make a pessimistic assumption: even if it knows that a particular operation will hardly ever block, it has to execute that operation in a separate thread just in case it does. Threadlets allow near-optimal handling of such cases with even less effort than such threading, and that’s good.

It will be interesting, though, to see if threadlets tend to mask concurrent-programming errors. It’s not hard at all to imagine a program in which some threadlet code always executes in the original context in testing, and then does switch to a new thread in production. In my experience, “dual mode” code that can run on either one or multiple threads (or for that matter processors) almost never exhibits the same bugs in both modes. I almost think it might be useful to have a way of telling the threadlet subsystem to run threadlets in their own threads even if it doesn’t need to, just to make sure that that case gets tested and debugged.

Foxtrot Hiatus

There’s a pair of words you don’t see together too often. One of the Big Deals in the comics world is apparently that Bill Amend, of the excellent and famous strip Foxtrot, is moving to a Sunday-only schedule. Pearls Before Swine (currently my favorite comic) has a strip about it today. It’s fairly amusing in an obvious way, but it also embeds a second joke that might be even funnier. In the third panel, Rat says “wow” … but if the dialog weren’t in all caps you’d probably see it as “WoW” instead. WoW is, of course, World of Warcraft, and rumor has it that WoW addiction is the real reason behind Amend’s change of schedule. I have no idea if the rumor is true, maybe I’m guilty of repeating something Amend would find offensive, but I still find the subtle mention in Pb4S pretty amusing.

More Fun Amy Sayings

In no particular order, just so I’ll remember these years from now …

  • “Snow seeds” (what most of us would call snow flakes)
  • “Daddy’s beautifuling the snow” (referring to the sprays of snow as I shoveled the driveway)
  • “Platypus on (just about anything)”
  • “Clock without pancakes” (because there are pancakes on the clock in Skip To My Lou)
  • “Don’t eat the otter”
  • “D is for Daddy” (spontaneous and not from a book, as far as I know)
  • “Read BLEEP” (instead of “read, please” and no, I don’t know where this one came from)

I’m also pleased to report that Amy can read (or really recite) most of some books all by herself right now. For example, she did Barnyard Dance last night, with help from me in only a couple of places. She can also do this entire section from Too Many Frogs all by herself, which I find quite impressive.

“But I do mind, Froggy. “I never invited you in,” Rabbit explained, “I never invited you to fix a snack, I never invited you to get all comfy-cozy, and I never invited your family to join you. So I do mind, very much indeed”
“Uh oh, Froggy said, “this will never do. Thanks for your kindness. Toodle-oo.

That’s a long passage for a two-year-old. Heck, I probably didn’t manage to get it quite right myself, from memory, but Amy does. Just when I think she’s not progressing as fast as she used to, she surprises me. She has also learned to jump horizontally, and vertically from a step-stool. Any day now, she’ll be jumping vertically.

UPDATE: Cindy mentioned another good one last night – “ramp shoes” referring to high heels. Also, “toe-thumb” referring to her big toe.

Sloppy Programming

Here’s a quick quiz. Let’s say that you have a Linux kernel module, built against the same version of the kernel that you’re running but with one slightly different configuration option, and you try to load it. It will:

  1. Work correctly.
  2. Fail to load, because it is recognized as being incompatible.
  3. Load but fail to function, producing some error messages that explain the problem.
  4. Crash your system in an utterly non-obvious way.

The answer, of course, is D. It shouldn’t be, but it is, at least if what you’re loading is a network driver and the configuration option is netfilters. The reason is simply bad software engineering, similar to another example I wrote about seven years ago. Both of the examples I mention there have been fixed, by the way; I guess that senior developer I mentioned managed to put his new knowledge about pointers and caches into context enough to do the right thing (or allow someone else to) at some point. Without getting too far into the technical details, because that’s not the point I want to make, the problem in this case is that turning on netfilters in your kernel adds fields to the sk_buff structure – one universally needed by any networking code – and adds them in the middle. Thus, inline functions in network drivers try to modify some common fields and actually end up trashing the structure. I thought everyone knew that, at the very least, new or optional fields should be added at the end of a structure. Better still would be to put these special-purpose fields in a separate structure, allocated along with the sk_buff itself or hidden in an skb_reserve area (as is done in some similar cases). Structure versions would be nice too, so at least a mismatch could fail gracefully instead of messily.

Of course, some Linux hackers will say such things are just aesthetic, because you have the source and should always build against the exact same kernel you’re using anyway. Bull. For one thing, this doesn’t affect only developers but users as well. I might have the source, but Joe User might not. If I therefore have to provide Joe with binaries, I now need to provide exactly the right binaries for every possible configuration Joe might be using, and that’s a logistical nightmare. If anyone wants people like Joe to use Linux, they should consider what makes life easier for people like Joe, and this kind of silliness definitely doesn’t. Incompatibilities involving an integral part of an API that’s used by hundreds of packages maintained by thousands of different people should simply be handled better, and not introduced spuriously just because someone wants a few new fields for their own use and Linux-kernel politics preclude adding them the right way. If Linux is a private toy for a closed community you can do that. If it’s something you expect to gain broad acceptance you can’t.

Funny Amy Tricks

Amy often has a phrase that just cracks her up and that she loves to repeat, especially at bed time. For a while it was “great big pretzel” from one of the Richard Scarry books. More recently it has been “Amy’s bread” which has had Cindy and me totally mystified. Two nights ago, the secret was revealed. In the middle of the evening, Amy said, “Amy’s bread” and I had the inspiration to ask, “Where is Amy’s bread?” She trotted over to the bread machine which sits in a corner of the dining room, opened it up, and took out a stuffed monkey toy that she apparently keeps there. Now “Amy baked a monkey” is also good for a giggle. You just never know what phrases are going to become common around your house when you have kids.

In other baby-related news, congratulations to Cathy and Wilson on the birth of Colette Adeline. Welcome to the club!

Second Mover Advantage

In my article about Revivio, I mentioned the “second mover advantage” – i.e. the phenomenon of the “first mover” expending all of their efforts breaking down barriers that they then lack the energy to cross themselves. I had thought of using the analogy of a soldier lying down on barbed wire so that others can cross it, but Bryce at work came up with an even better metaphor.

The early bird gets the worm, but the second mouse gets the cheese.

There is a story behind it, but I think it’s worth pondering a bit first.

Excitement in Boston

By now, most of you have probably heard about (or been affected by) the discovery of several “devices” in various places around Boston. They turned out to be either a harmless prank (includes image) or a guerilla marketing campaign. Mark Frauenfelder at Boing Boing has been all over this, as the devices were reported as having been based on blueprints from Make magazine. Mark and his readers are outraged.

Ashamed of their own foolish overreaction to finding some battery powered Lite Brites and promptly declaring a code red emergency that shut down Boston, authorities are hoping to save face by arresting Peter Berdovsky, who is charged with installing the innocuous signs.

Sorry, Mark. I love the stuff you post, and I understand that this kind of stuff is just innocent fun, and usually it is, but this time it’s not. The authorities do have to respond this way when they find electronic devices under bridges and stuff. Bombs can have cutesy LEDs too. It was probably not fun at all to people who were late to work or important meetings, parents who missed seeing their children that day, etc. It would have been really not funny if emergency vehicles had been blocked or delayed from getting where they needed to go. Sorry your hobby got caught in the crossfire, but you have no right to feel aggrieved at the authorities. Blame the morons who showed no sense or consideration at all when they did this.