Yet another side topic that has come up in recent conversations is that of code layering. In a way, it’s amazing that the idea would even need to be discussed with anyone past about their sophomore year in CS, but apparently it does. I wouldn’t expect anyone to say that modularity is stupid, or that “divide and conquer” was a dumb idea, but someone actually did trot out the old “layering is for cakes, not software” brainturd the other day. Leaving aside that I prefer unlayered cakes, I responded thus:

It’s a good thing there are smarter people than you, who don’t think in simplistic slogans. Collapsing layers is a common trick in the embedded world, to wring every last bit of performance out of a CPU- or memory-constrained system at cost of adaptability. In a system that is constrained by neither CPU nor memory and where future extension should be expected, it’s a bad approach. If networking folks had thought as you do, it would be a lot harder to implement or deploy IPv6 or the current crop of transport protocols, not to mention packet filters and such that insert themselves between layers. In other words, it would be harder for technology to progress. In storage that same story has been played out with SCSI/FC and PATA/SATA/SAS.

Anybody who has done serious work in networking knows that network-layer and transport-layer expertise are rarely combined in one person, and that there are several other areas of expertise both above and below those two. In storage a good team will often have some people who know the low-level Fibre Channel glop and others who know the (slightly) higher-level SCSI-command-set stuff. There are HBA-driver developers and generic-disk-driver developers and volume-manager developers and filesystem developers, again with more above and below. Why? Because there’s enough complexity in any one of these areas to keep even a sharp mind fully occupied. One of the most important aspects of layering, besides its obvious necessity in making it possible to test software properly, is that it allows each area to be covered by an expert in that area. It allows all of that expertise to be combined effectively not only within an organization but across organizations as well. It’s like that old joke:

Heaven is …

Where the police are British,
The cooks are Italian,
The mechanics are German,
The lovers are French and
It’s all organized by the Swiss.

Hell is …

Where the police are German,
The cooks are British,
The Mechanics are French,
The lovers are Swiss and
It’s all organized by the Italians.

Here the roles are things like filesystems and volume managers while the nationalities are companies like Veritas and Oracle, but you get the idea. Of course, the problem is that people within each specialty tend to view the others with something akin to contempt. Jeff Bonwick wrote about SANs being designed by firmware engineers, with the obvious implication that such people could never do the job right. I’ve never been a firmware engineer but I’ve known a few, and they’re pretty likely to view what Jeff and I do with equal contempt – with just as much reason. Meanwhile, a database-server designer at Oracle would likely look down their nose at the whole stinking lot of us. In fact, Oracle is notorious for pretending that the world would be better off without things like filesystems, or even operating systems, at all. After all, only the database itself really knows what its needs are, what data should be kept around, how activities should be scheduled, etc. Information gets lost every time you cross from the database server to the operating system, and whenever the latter tries to do anything it’s likely to just get in the way.

If that reasoning sounds familiar, it should. It’s exactly the same refrain we’ve been hearing about why a clean interface between filesystems and volume managers is a bad idea, which led to the brainturd I mentioned earlier. If it’s such a great idea for a filesystem developer to implement volume-manager functionality, why isn’t it also a good idea for a database developer to replace parts of the operating system that they don’t like? Is it because layering is suddenly a good idea again when it’s layering that suits the filesystem developer, or is it because filesystem developers are just so much smarter than everyone else? Now, I’ve been accused of arrogance. I think I’m pretty darn good at a few things, and competent in a whole lot more, but even so I’m not arrogant enough to think I know better than everyone else even within their own areas of expertise. It’s possible that a development group filled with British-cook wannabes can succeed, but I think “people doing what they’re best at” is a more likely recipe for success.