Hack the Planet linked to an interesting MS Research project called Singularity. From the blurb:

Singularity is a research project in Microsoft Research that started with the question: what would a software platform look like if it was designed from scratch with the primary goal of dependability? Singularity is working to answer this question by building on advances in programming languages and tools to develop a new system architecture and operating system (named Singularity), with the aim of producing a more robust and dependable software platform. Singularity demonstrates the practicality of new technologies and architectural decisions, which should lead to the construction of more robust and dependable systems.

The result (so far) looks pretty interesting, particularly in light of my own recent comments about microkernels. Their design is definitely that of a microkernel, with components represented by Software Isolated Processes or SIPs, which run in a single address space. The kernel (and, I assume, a core set of SIPs) is written in a “safe” language called Sing#, which is derived from another MSR project called Spec#. In fact, leveraging other MSR projects seems to be quite a theme in Singularity; they also use the Boxwood filesystem, for example. Spec# looks like a very interesting language, with built-in support for verification of program properties way beyond the mere type-checking you find in most languages. There are specific features for null-pointer handling, safe exceptions, and detailed specification of Eiffel-style “contracts” between functions. The specific difference between Sing# and Spec# is that it extends this validation even to protocol-level interactions between components (e.g. “if I send you an X message I expect either a Y or Z in response”) which is very cool. To a large degree these features are implemented via static code analysis rather than run-time checks, so the impact on performance should be small.

In many ways Singularity looks like the kind of OS I might have designed. They pretty much prove my point about a microkernel design implemented within a single address space with lightweight RPC, for starters. Their work also addresses my point about safe languages. Such languages are great, but actually implementing a kernel in an interpreted/JIT-compiled fashion would be insane. Singularity’s approach of relying on a big dose of static verification to enforce safety properties in code that then runs in fully compiled form seems like very much the right way to go about things, and the limitations placed on SIPs make perfect sense in a context of keeping the system reliable.

There are only two things that I would consider disappointing about Singularity. If they’re going to all this trouble to create an OS geared toward dependability, it seems odd that they’re not more actively working toward a distributed system that can survive hardware as well as software faults. Maybe they just decided that the scope of the project was large enough already and they’ll leave that part for later, which is fine, but it seems like it should merit at least a mention. The other thing is not so much a disappointment as an area where further improvement can be made. The protocol-level checks in Sing# are great, but incomplete in two ways. One, which the authors admit, is that the verifier checks for invalid state transitions but not deadlock/livelock. The other is that, while the protocol contract addresses the issue of running but possibly malfunctioning components, it does little to address what should (or should not happen) when a component outright fails and is removed from the system. The behavior of other components that were interacting with the failed one should also be specified and enforced, though the exact details of that specification are obviously up for debate.

Quibbles aside, though, Singularity looks like a very interesting piece of work. I look forward to seeing what directions they take it from here.

Shareholder Value

Mark Kleiman has posted an excellent response to the notion – championed most (in)famously by Milton Friedman – that a company’s sole moral obligation is to make profit for its shareholders. His approach is reductio ad absurdum, but I think there’s an even more important flaw in Friedman’s reasoning. This flaw is the all too common assumption that “money is everything” and therefore any value not represented in monetary form is irrelevant. In this case, this leads to believing that people invest only based on (direct, short-term) monetary return, but that’s simply not true. When people buy stock, they do so based on a certain assumptions. They assume that certain legal and moral restrictions are applicable to what the company does, and they invest based on that assumption. This is particularly true of “green” or socially-conscious investors, who might be making decisions based as much on a company’s image or reputation for ethical behavior as on their purely financial performance. In a sense one might say that such investors have monetized their morals by making such investments, but that doesn’t mean they’ve given up those morals forever in return for profit. Presenting such an image and then acting in a wholly different manner is a form of fraud, and unconscionable. The same principle applies to every company and investor, though usually to a lesser degree. If the moral justification for what companies do is fulfillment of shareholder expectations, then expectations other than profit must be considered.

There’s an even more fundamental problem that shareholders do not adequately represent the interests of all who are affected by a company’s actions, and that those others deserve consideration too, but that’s probably best left for a future article.

Update: Mark’s trackbacks seem to be broken, but Michael O’Hare (writing on Mark’s site) seems to echo my point rather well.


Apparently it’s made up (I particularly like the last sentence), but it’s a darn good word nonetheless. I think I’ll try using it at work.

On the Internet…

…nobody’s knows you’re a grunt. If you were to judge by the claims made on Slashdot and similar forums, you’d think that architects and consultants with decades of experience are a dime a dozen, and novice programmers as rare as hen’s teeth. Usually such fraud is merely annoying, but sometimes the cognitive dissonance between someone’s statements and their claimed credentials is downright hilarious. As Exhibit A, I offer never nest loops:

I have for the last 10 years been employed in various places as a senior architect/mentor, to instruct and set coding policy across the enterprise..

And I would shoot anyone who nested a loop under any circumstances – let alone three levels, and especially if you plan on exiting from it!

Loops come in two forms : processing loops (ie for every value we want to do something) – and search loops (we want to go through until we find something)

What a great idea. Because our “senior architect/mentor” is uncomfortable interpreting even simple control structures except in splendid isolation, we should adopt a rule that forces us to:

  • break up even fairly simple logic into multiple functions, destroying any readability it once had;
  • force local variables to be passed as arguments and return values or (even dirtier) member variables;
  • rob “return” of a potentially useful meaning, either forcing return values to be propagated through N levels or relying on exceptions which are less efficient and might not be viable for your environment;
  • abuse the stack to hold all of those arguments, return values, and function/exception frames;
  • risk subtle bugs that occur when an argument (which used to be a local variable) is modified and not propagated back to the caller (which used to be part of the same function).

I’m sure I could think of a half-dozen more reasons not to take Darren Oakey’s advice, but the point should be clear. Even if his suggestion had technical merit, it would undoubtedly cause a rebellion among the programming staff — a fact I’m sure any real architect or mentor would know. I guess it should come as no surprise that Darren also hates specs. Somehow I don’t think most employers are rushing to hire architects with that attitude. Here’s a rule that might really be useful for net.experts: if you’re going to play a role, at least try to be convincing.

ZFS Again

It looks like there’s a new set of sickeningly cute and self-congratulatory slides for Sun’s ZFS, about which I’ve written before. At least one of my predictions seems to have come true. If you look at slide 24 (“Object-Based Storage”) it’s quite clear that there really is a volume manager lurking in there, even though they try to play on people’s fears of volume managers and make it seem that ZFS has made the need for them go away. Personally, I think that such misleading claims are a blot on an otherwise excellent project. There’s a lot of great stuff in ZFS, from streamlined management to data-integrity to performance and scalability (though I do wonder why someone supposedly involved in “SAN Engineering Product Development” doesn’t seem bothered by ZFS not being a true SAN filesystem) but if I were one of the engineers on that project I’d really be getting on marketing’s case about how it’s (mis)represented.

Enough about that, though. I already wrote about that last time. What’s nearer and dearer to my heart this time, since I’m working in the Continuous Data Protection space, is the claims they make about nearly infinite and nearly free snapshots in ZFS. It makes me wonder how they deal with space reclamation, and of course those issues aren’t even mentioned in any of the slides etc. I’ve seen. If you’re keeping old copies of data around, your filesystem is eventually going to fill up. Sooner or later, no matter how much space you have allocated, you’re going to need to toss out some old data to make room for new; it’s not very nice to start throwing errors to whoever’s writing the new data because the disk is full of months-old data in which nobody has even expressed an interest recently. If each block could potentially be referenced as part of the active filesystem and as part of several snapshots/clones, how do you know when it’s safe to get rid of it? Refcounts? Think for a moment about how important it is to make sure those refcounts are fully up to date with every transaction, and the potential performance cost of making sure that they are. If, on the other hand, each block can belong to only one “view” (active/snapshot/clone) of the filesystem, then massive copying is necessary whenever a snapshot/clone is created and they could hardly be as instant or as infinite as they’re claimed to be. I’m not saying clever approaches don’t exist to ameliorate this problem – how we deal with it is an important part of our “special sauce” at Revivio – but it is most definitely a core problem for this kind of system and one that anyone bragging about capabilities should at least mention.


A recent Slashdot article about the latest release of Tanenbaum’s Minix led, predictably, to a rehash of the old microkernel vs. monolithic kernel flame war. All of the usual non-fun ensued, such as having to point to L4 and QNX to refute myths about how microkernels are necessarily slow and have never proven useful in the real world. The weirdest part was when a couple of microkernel-bashers who clearly had no idea how much OS support they get for their own programs tried to suggest that the real solution to the world’s problems was to write kernels in a modern “managed code” environment such as Java’s JVM or .NET’s CLR. My observation that they had just proposed a different kind of microkernel was met with sullen silence.

Microkernels are, and always have been, a good way to design operating systems. Note that I said design. Actually implementing microkernels as collections of separate heavyweight processes communicating via heavyweight IPC is not such a good idea. Like distributed filesystems (another technical area that is finally receiving some long-overdue attention), poor early implementations of a basically sound idea led the less informed to dismiss the idea itself. For a microkernel to be usable, the process/IPC model needs to be such that communication between components is efficient. In an extreme form, this could even mean that the actual delivered implementation of a microkernel-based OS actually has all of those “processes” in a single address space with IPC resolved down to simple function calls or even inline code. None of that makes the result any less of a microkernel design. The defining characteristic of a microkernel is not the use of multiple address spaces but the idea that a bare minimum of functionality should not be hot-swappable. By hot-swappable I mean that anything other than the microkernel itself should be separately loadable and unloadable and restartable, and to at least some extent should be able to fail without bringing down others. At a minimum this precludes a global panic() call. Generally it should also include an inter-component communications and error-handling model that prevents one component from indefinitely and explicitly waiting for another which has failed (though implicit waits such as those I describe at the end of my explicit state article are likely to remain a problem solvable only through programmer discipline). As I pointed out in the Slashdot thread, just having loadable/unloadable kernel modules is an important step in this direction and away from truly monolithic kernels, and for many good reasons it’s a step that even the anti-microkernel Linux made.

Modularity and maintainability are sufficient justifications for microkernel design, but they also have another advantage that I never got around to mentioning in the Slashdot thread: microkernels are easier to make distributed. It’s simply easier to distribute components that already use a defined IPC mechanism than those that use an incestuous “fat” interface involving all manner of calls and callbacks and twiddling each other’s bits. QNX is the premier example of this, achieving fault-tolerance levels of which Linux will simply never be capable. MOSIX is certainly good for providing an illusion of full distribution at the system-call level, but it’s a bit hacky and the illusion is never quite the same as a real distributed OS. Distributed OSes were, of course, one of Tanenbaum’s major interests, which might explain why he and Linus differ on the value of microkernels. Just as he once thought nobody needed SMP, and still thinks debuggers or specs are superfluous, Linus probably doesn’t see the value in distributed OSes and therefore in design approaches that facilitate them. The computing community in general should not be bound by one person’s short-sightedness, though. Microkernels are no magic bullet, but they have proven themselves as a good design approach for modern operating systems.

Smart Investors? Or Crooks?

Is anyone surprised that senators consistently beat the stock market? What’s really sad is that nobody should be.

Last year, Alan Ziobrowski, a professor at Georgia State, headed the first-ever systematic study of politicians as investors. Ziobrowski and his colleagues looked at six thousand stock transactions made by senators between 1993 and 1998. Over that time, senators beat the market, on average, by twelve per cent annually. Since a mutual-fund manager who beats the market by two or three per cent a year is considered a genius, the politiciansâ?? ability to foresee the future seems practically divine. They did an especially good job of picking up stocks at just the right time; their buys were typically flat before they bought them, but beat the market by thirty per cent, on average, in the year after.

Being an Architect

Dan Sugalski, of Parrot fame, has written a post mortem regarding his experiences running that project. Some of what he says struck a chord with me, so I thought I’d elaborate on it from my own perspective.

Professional courtesy: Good
One of the things I insisted on was that people behave professionally. (Not, mind, as adults — I know plenty of adults who behave badly, and some of the kids (and yes, Brent, you and Zach counted at the time :P) involved with parrot behaved better than some of the older adults) I wasn’t looking for the sort of cult of personality that pervades the perl, python, and ruby development camps — I find that rather distasteful, both as the potential target and as a general participant. I also remembered the perl5-porters mailing list when it was more a shark tank than a development mailing list, and I did not want that.

All-volunteer or not, I viewed parrot as a professional engineering project, and I wanted people to behave accordingly. For the most part everyone did, and I was happy about that.

Not telling people to shut the fsck up and cope: Bad
Professional courtesy is fine and all, but if you’re running a project, with volunteer labor or not, sometimes you need to tell people to suck it up and deal. More importantly, there’s nothing wrong with a judicious exercise of authority, especially if the exercise of authority is part of the job you hold. I was the architect for the project and it was my responsibility to design the system and get that design implemented. Part of that meant that yes, I should have told people to cope. Shying away from confrontation doesn’t do anyone any good — I did, and the project suffered for it.

This is the part that I also found most difficult about being the product architect at Revivio, and is to a large degree why I’m not in that role any more. A lot of people have this idea that being an architect means you call all the shots, which sounds pretty cool. I used to think that myself, but the reality is quite different. The fact is that everyone’s much more willing to share responsibility with the architect, but not control, so the architect ends up being held responsible for a lot of decisions he did not in fact make. Nobody has the time to review every line of code that gets written by every developer on any non-trivial project, and if you’re not writing or reviewing the code you don’t really have control over how it’s implemented. I can’t count the number of times I worked my butt off to secure some kind of compromise on how something should work, only to have the developers go off and implement something distinctly inferior to what had been agreed upon. Sometimes the exact problems we had anticipated and tried to avoid in the design phase came back and bit us as last minute bugs because of that. To this day, the same person responsible for one particular piece of brokenness is now lobbying for permission to rewrite his pet subsystem for about the fourth time because of flaws that he had been told about well over a year ago. It’s pretty frustrating.

The worst variant of this problem is what I call guerilla warfare. We used to have our own “Leo” here, and if contributing to his departure had been the only thing I ever did as architect it still would have been worth it. His specialty was to try to undo every decision he didn’t like. Sometimes he’d simply go off and do something different, as described above. Sometimes he’d lie about having received permission or agreement for it. Most often, though, he’d just take every opportunity to question both the original decision and the authority/competence of the person who made it. I’m sure everyone else is familiar with this phenomenon. Every single bug, no matter how minor, in a component that was once the subject of controversy becomes an excuse to reopen the controversy. “There’s a typo in an error message? Oh, well, that’s proof positive that we should have done it my way. Never mind how far we are into the project; let’s rewrite it now.” Insert eye-roll here. Fortunately, management eventually wised up to these tricks and demanded that they stop…not before enormous damage had been done both to the code and to morale, but nonetheless.

It’s easy to say that some of these problems might have been less severe if I had put my foot down earlier, but there’s only so much you can do even in a professional environment without harming morale and productivity even more. If you’re not responsible for assigning someone’s priorities or writing their review, you actually have very little control over what they do. If the director of development resists giving you authority to reject checkins on the basis of technical inconsistency or insufficiency, another means of ensuring technical quality is lost. There’s nothing quite like being relegated to a purely advisory capacity, where others can pretty much choose to ignore your advice, and then being held accountable for the consequences when they do. This is but one instance of what I generally consider Management Mistake #1, particularly at startups: failing to associate authority with accountability. The only thing worse than blaming someone for something they couldn’t control is giving someone (usually senior management) control over something for which they’ll never be held accountable. If you’re going to let a VP make technical decisions, they have to be held accountable for any bugs or schedule slips attributable to those decisions. If you’re going to have one person as the “one neck to wring” for all technical aspects of a project, you have to give them some way to make that project reflect their vision and not somebody (or everybody) else’s. Some day I’ll be at another company, whether things go well or poorly where I am now. Whatever role I’m looking for, you’d better believe I’m going to ask a lot of questions about how authority and responsibility are distributed and whether the two match.

Meet the New Boss

Res Publica at Republic of Dogs has posted an excellent response to new-generation narcissism which, among other excellent points, pounds at one of my favorite memes (or perhaps anti-memes):

And here we meet Straw Man #2: Ideology vs. Pragmatism! This is perhaps the biggest and steamiest load of hork that has ever spurted up from the dyspeptic GI tract of the so-called New Democrats, only to be endlessly swirled around in the collective mouth of the Democratic Party like a fine wine, only with chunks.

There hasnâ??t been much seriously theoretical or idiological thinking in general circulation in our party for decades. Most Democrats, including progressives, are deeply pragmatic, but thereâ??s pragmatism and then thereâ??s pragmatism. Kos, it seems, thinks that political pragmatism consists of advocating whatever policy sounds pretty darn good to folks. I would argue that the better form of political pragmatism, the one that actually represents the larger Pragmatist tradition in American political philosophy, consists of advocating policies that we have good reason to believe will work, and then present our fellow citizens with reasonable arguments to persuade them to support our proposed policies. But I guess I should just save that talk for Old School Activist back at the commune.

I donâ??t think the arguments that have been made against vouchers are particularly ideological. While it was adequately funded, Americaâ??s system of public schools transformed the social landscape. Universal public education encouraged social mobility and helped to expand and stabilize the American middle class. Last time I checked, Democrats supported opportunities for upward class mobility, and believed that a stable middle class helps democracy thrive. To allow conservatives to under-fund the public schools of poor children and then rush in to deliver the systemâ??s death-blow with their cockamamie privatizations and voucher schemes isnâ??t pragmatic. Itâ??s just dumb. Itâ??s a purely ideological decision that universal public education for effective citizenship isnâ??t important, and that the state should provide poor schools for poor people or leave the matter up to Jesus & Friends. It is, in short, an obvious example of the thinking of blow-dried neo-liberals who have drunk deeply at the ideological wells of the new capitalism — ” the kind of people who use words like â??proactivenessâ?.

This is exactly the same point I’ve been trying to make for years, seemingly to no avail. Political/economic ideology represents a strategy to achieve pragmatic ends. I don’t choose a so-called liberal ideology or social policy because it’s more beautiful or more “right” in some mushy abstract way; I choose it because I believe it leads to better practical results. Call it true enlightened self-interest, based on knowledge of game theory and chaos/complexity theory and good old-fashioned economics (the kind that’s actually based in reality instead of attempting to impose its own dogma on reality). Many people just don’t understand those kinds of things, so they fall into the various conceptual traps based on ignorance of how things work (and people behave) in the real world — a category that includes both Marxism and laissez-faire capitalism. Maybe Schelling’s Nobel prize for his work on cooperation will help with that, but I don’t hold out much hope; he’s not the first to receive a Nobel for work in this area, but too many still seem to understand competition about ten times better than cooperation and value it a hundred times more. As Res Publica points out, the distinction between ideology and pragmatism is bogus. It’s usually made as part of an attempt by those on the right to frame those on the left as starry-eyed dreamers unconnected to reality. For the “new left” to accept the right’s caricature amounts to betrayal of their own putative cause. Instead, we should point out that the flip side of being a starry-eyed dreamer is being a short-sighted groveler too absorbed in the pursuit of narrow self-interest to notice the disaster looming. I’d rather be Jonathan Livingston Seagull than a vaguely possum-shaped piece of hamburger and fur on the highway. Does that sound pragmatic to you? Idealism is a good thing, as is intellectualism. They both serve pragmatic goals better than blind faith in an internally inconsistent dogma unconnected to empirical reality.

Fruit Sushi

For a while now I’ve been toying with the idea of making some kind of “fruit sushi” dessert/snack. Accordingly, tonight I embarked on my first experiment. The recipe is basically fruit rollups, rice-krispie square mix, and chopped dried fruit (conveniently provided by Sun-Maid in this case). Here’s a picture of the best-looking roll I managed.

fruit sushi

I learned several important lessons during tonight’s experiment.