Many Eyes

Yeah, yeah, I know, it’s been quiet around here. Between being in full-on crisis mode at work and having some sort of stomach-flu thing (two bugs at once, in a way) I haven’t had much time or energy left for writing. Both the crisis and the illness have abated enough for me to take a bit of a break now, so here goes.

One of the favorite myths of open-source advocates is the idea that “many eyes make all bugs shallow” – i.e. that if you just put something in front of enough people they’ll find all of the bugs for you. Besides the fact that it’s essentially the same “users as testers” attitude that many of those same advocates (incorrectly) accuse Microsoft of having, it’s just not true. What does “shallow” mean for a bug, anyway? The most intuitive meaning would be something that fails obviously and consistently in common circumstances. A “deep” bug, by contrast, would be one that fails only in unusual circumstances (e.g. an uncommon combination of hardware or sequence of operations/events), that does not fail consistently (e.g. if it’s timing-related), and/or that does not fail obviously (e.g. memory leakage or corruption which do not become apparent until much later). Those are the bugs that give developers nightmares. At Revivio, for example, we’ve had two bugs that we dubbed “Paperweight I” and “Paperweight II” because that was about the functionality left in our product when they happened. They both involved unusual circumstances. The first was not only timing-sensitive but also heat-sensitive, which is something software engineers don’t usually expect. The second was a kernel stack overflow caused by nested interrupts, which means it was not only very timing-sensitive but also that it led to symptoms that were only consistent in occurring long after the actual overflow itself. Those are “deep” bugs.

The key point here is that the “depth” of a bug is intrinsic and does not change according to who’s looking. Many eyes do nothing to affect which bugs are shallow and which ones are deep, but only which ones are found and fixed; “most eyes see only shallow bugs” would be a lot closer to the truth. Deep bugs are (at least) as real and worthy of our consideration as shallow bugs, but finding them before they actually cause something Very Bad to happen requires more than casual use. The breadth of testing that comes from having many users is valuable, but is also no substitute for the depth of testing that comes from trained people using rigorous methods to ensure that every boundary condition is tested. No number of generic college students walking around on a patch of land will be as good at finding oil as one trained petroleum geologist equipped with seismic testing equipment and big computers to analyze the results. Metaphorically speaking, software companies hire that geologist, provide her with that equipment, and pay her to go where she might not otherwise. Open-source projects are more like the masses of generic college students. They’ll all cover the same ground over and over again for ever-decreasing benefit, and practically none will probe any further until it’s too late – i.e. until the bug has already bitten someone, and quite likely until other code has been layered on top of the bug making it harder to fix. To the extent that there are many eyes at all, given the fact that there seem to be at least a dozen separate open-source projects to do just about anything, they often don’t do much to affect software quality.

Civ4 First Impressions

The title should tell most of my readers everything they need to know about why I haven’t been writing much lately. I’ve been a fan of the Civilization series for a long time, having played all four of the main titles plus Alpha Centauri (including Alien Crossfire) and a few inferior knockoffs from other development shops. Civ2 will probably always reign as the best game for its time. It fixed some of the major combat-system warts in the original, such as spearmen defeating tanks, and added a whole bunch of other neat stuff. Spies were more fun in Civ2 than in its successors, and there were several other interesting units. I was always fond of paratroopers as instant city garrisons, and wonder why they seem to be absent from Civ4. Civ2 also made diplomacy more interesting, though the “everyone hates you” phenomenon was still gallingly prevalent, and having to manage happiness as well as money presented an interesting challenge. Taking a slight detour, Alpha Centauri’s “design workshop” that let you design your own units based on components absolutely rocked, and is still unique among the group. AC also had a few other conceptual improvements that I’ll get to later, and might be the best game overall. Civ3 had diplomacy that actually kind of worked, enemy AI that wasn’t totally stupid, and the new dimensions of resources and culture. It almost seems like it should have been better than Civ2, but somehow I just never got into it as much. That’s about enough history, though. What about Civ4?

Iraqi Adventure

Iraq as a game of Adventure. Brilliant! I know it’s political and I’ve been trying to get away from that here, but it’s not quite serious enough for It Affects You. It also contains some “inappropriate language” in case anyone cares.

Petri Net Test Generator

The purpose of PNTG (pronounced “painting”) is to provide an easy way to generate test functions for code with discontinuous control flow, such as a network protocol where sections of code are separated by waits for messages or an API in which users call multiple entry points in succession. The resulting test functions can then either be compiled and run as a standard unit test or subjected to static code analysis. Either way, the goal is to verify that the code follows certain rules such as locking or resource acquisition/release across all possible sequences — including those which are likely to be missed by manual test generation and almost certain to be missed by static analyzers on their own.

Note: this article and the ideas behind it are not really “fully baked” by my usual standards, but I’ve been falling behind on my writing here so I figured I’d just splat it up here anyway to see if anyone had any interesting comments. Maybe I’ll try to clean it up more later.

Funny Headlines

This just had to be deliberate. Successive headlines in the “world” section of the Sydney Morning Herald today:

Cannibal faces new trial
Senate roasts Supreme Court nominee

Code Reviews

Jeff Kesselman (a.k.a. gameguy) at Sun posted about why he doesn’t believe in code reviews. I responded taking the contrary view that code reviews can be worthwhile (even though they often aren’t). The result is excerpted over the fold so that my readers as well as his can see it and join the debate.

More Pictures

OK, time for more Amy pictures, this time from Christmas until New Year.

2006

As I’m sure everyone must have assumed (or known) I’ve been out of town and out of touch for the last week, visiting family in Michigan. I could actually have spent more time online if I had wanted, because they do have internet access both in Ann Arbor and in Batavia NY where we stayed overnight in both directions, but this is my opportunity to wean myself off the electronic teat and start the new year fresh so I (mostly) took advantage of it. There’s not much to report, really. The only interesting thing about the travel part was the absolutely ginormous room the hotel gave us on the way back. I’m no novice traveler, but I still had to double check that we’d gotten the right room/rate because nobody expects a ~1000 square foot mirrors-everywhere – yes, even over the bed – suite for under $100 a night. Amy had a blast toddling around all that empty space, but it was actually not as nice a room as the one we’d had on the way out in terms of finding a light- and sound-isolated place for her to sleep. I ended up reading a book by flashlight in the furthest corner of the room while she settled down. While in Michigan, I had the usual moments when I remembered why it’s good to have a family, and other moments when I didn’t. All in all, everything went much as expected.

I do have some pictures and video from Christmas and the trip, but I haven’t had a chance to process them yet. Instead, here are some pictures that I hadn’t put up yet from before.