Yeah, yeah, I know, it’s been quiet around here. Between being in full-on crisis mode at work and having some sort of stomach-flu thing (two bugs at once, in a way) I haven’t had much time or energy left for writing. Both the crisis and the illness have abated enough for me to take a bit of a break now, so here goes.

One of the favorite myths of open-source advocates is the idea that “many eyes make all bugs shallow” – i.e. that if you just put something in front of enough people they’ll find all of the bugs for you. Besides the fact that it’s essentially the same “users as testers” attitude that many of those same advocates (incorrectly) accuse Microsoft of having, it’s just not true. What does “shallow” mean for a bug, anyway? The most intuitive meaning would be something that fails obviously and consistently in common circumstances. A “deep” bug, by contrast, would be one that fails only in unusual circumstances (e.g. an uncommon combination of hardware or sequence of operations/events), that does not fail consistently (e.g. if it’s timing-related), and/or that does not fail obviously (e.g. memory leakage or corruption which do not become apparent until much later). Those are the bugs that give developers nightmares. At Revivio, for example, we’ve had two bugs that we dubbed “Paperweight I” and “Paperweight II” because that was about the functionality left in our product when they happened. They both involved unusual circumstances. The first was not only timing-sensitive but also heat-sensitive, which is something software engineers don’t usually expect. The second was a kernel stack overflow caused by nested interrupts, which means it was not only very timing-sensitive but also that it led to symptoms that were only consistent in occurring long after the actual overflow itself. Those are “deep” bugs.

The key point here is that the “depth” of a bug is intrinsic and does not change according to who’s looking. Many eyes do nothing to affect which bugs are shallow and which ones are deep, but only which ones are found and fixed; “most eyes see only shallow bugs” would be a lot closer to the truth. Deep bugs are (at least) as real and worthy of our consideration as shallow bugs, but finding them before they actually cause something Very Bad to happen requires more than casual use. The breadth of testing that comes from having many users is valuable, but is also no substitute for the depth of testing that comes from trained people using rigorous methods to ensure that every boundary condition is tested. No number of generic college students walking around on a patch of land will be as good at finding oil as one trained petroleum geologist equipped with seismic testing equipment and big computers to analyze the results. Metaphorically speaking, software companies hire that geologist, provide her with that equipment, and pay her to go where she might not otherwise. Open-source projects are more like the masses of generic college students. They’ll all cover the same ground over and over again for ever-decreasing benefit, and practically none will probe any further until it’s too late – i.e. until the bug has already bitten someone, and quite likely until other code has been layered on top of the bug making it harder to fix. To the extent that there are many eyes at all, given the fact that there seem to be at least a dozen separate open-source projects to do just about anything, they often don’t do much to affect software quality.