Canned Platypus

I love programming. I love thinking about algorithms and data strucures. I love writing code, rearranging code, talking about code. I even love testing and debugging and documenting code. (This is not to say I do all of these things as consistently as I can. There are still only 24 hours in a day and so one must prioritize.) Sometimes I think of getting out of this field, though, because so much of working as a programmer nowadays has nothing to do with any of the things I love, and it seems to be getting worse. Nobody loves meetings and bureaucracy and such, but that’s not what I’m talking about.

I hate spending half my time dealing with build systems, source-control systems, package managers, and such. There are too many out there, they all suck, everybody has their favorite one and their favorite way of using it, and they’re not at all shy about ramming their preferences down your throat . . . which brings me to my real point. I hate programmers. Hot damn, but we are a noxious breed, aren’t we? I’m tired of the backstabbing, the trashing each others’ work, the holier-than-thou attitude from the GNU types, the rampant sexism, the bike-shedding, the endless effort to do and re-do all the fun stuff while dumping as much work as possible onto one’s peers, and on and on and on. I know I’ve exemplified many of these sins myself, I don’t need anyone else to tell me that, but if I made it my life’s goal to be as much of a jerk as possible I’d still find myself outdone just about every day by people who aren’t even at their worst.

Of course, I don’t know what else I’d do that pays the bills, so you all are stuck with me, but come on, people. Let’s stop sucking all the fun out of this profession.

As many people I’ve talked to IRL probably know, I really hate language-specific package managers. Java has several, Python/Ruby/Erlang etc. each have their own, etc. I totally understand the temptation. I know it’s not all about NIH Syndrome (though some is); some of it’s about Getting Stuff Done as well. Consider the following example. I tried to install Tornado using yum.

[root@fserver-1 repo]# yum install python-tornado
Loaded plugins: presto
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package python-tornado.noarch 0:1.0-2.fc15 set to be updated

(hundreds of lines of dependency stuff)

Transaction Summary
=====================================================================================
Install      13 Package(s)
Upgrade     161 Package(s)

Total download size: 156 M
Is this ok [y/N]: n

Is this OK? Are you kidding? Of course it’s not OK, especially when I can see that the list includes things like gcc, vim, and yum itself. I know how systems get broken, and that’s it. By way of contrast, let’s see how it goes with easy_install.

[root@fserver-1 repo]# easy_install tornado
Searching for tornado
Reading http://pypi.python.org/simple/tornado/
Reading http://www.tornadoweb.org/
Best match: tornado 1.0
Downloading http://github.com/downloads/facebook/tornado/tornado-1.0.tar.gz
Processing tornado-1.0.tar.gz
Running tornado-1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-6Wcauv/tornado-1.0/egg-dist-tmp-NEPqMm
warning: no files found matching '*.png' under directory 'demos'
zip_safe flag not set; analyzing archive contents...
tornado.autoreload: module references __file__
Adding tornado 1.0 to easy-install.pth file

Installed /usr/lib/python2.6/site-packages/tornado-1.0-py2.6.egg
Processing dependencies for tornado
Finished processing dependencies for tornado

Yeah, I see the appeal. On one hand, hours spent either rebuilding a broken system or debugging the problems that are inevitable when 161 packages get updated. On the other hand, Getting Stuff Done in about a minute. Yes, I tested, and the result does work fine with the packages/versions I already had. Still, though, having to do things this way is awful. It’s bad enough that there are still separate package managers for different Linux distros, but now programmers need to have several different package managers on one system just to install the libraries and utilities they need. Worse still, most of these language-specific package managers suck. None of them handle licensing, and few of them handle dependency resolution in any kind of sane way. One of the most popular Java package managers doesn’t even ask before downloading half the internet with no version or authenticity checking to speak of. Good-bye, repeatable builds. Hello, Trojan horses. I can see (above) the problems of having One Package Manager To Rule Them All, or of having dependency resolution be too strict, but there has to be a better way.

What if the system package manager could delegate to a language-specific package manager when appropriate (e.g. yum delegating to easy_install in my example)? Then the system package manager could save itself a lot of work in such cases, and also avoid violating the Principle of Least Surprise when installing in the “standard way” for the system yields different results than installing in the “standard way” for the language. There’d still be difficult cases when dependencies cross language barriers, but those are cases that the system package manager already has to deal with. I know there are a lot of details to work out (especially wrt a common format for communicating what’s wanted and what actually happened), possibly there’s even some fatal flaw in this approach, but my first guess is that a federation/delegation model is likely to be better than an everyone-conflicting model.

Presentations are the bane of the modern engineer’s existence. If you’re watching a presentation then it means you’re in a meeting, which is already something most of us don’t enjoy, and even worse it means you’re in a kind of meeting (or part of a meeting) that’s only minimally interactive. If you’re giving a presentation, that means even more time away from the technical tasks that drew you to this profession. Nonetheless, any project leader/advocate nowadays and for the last several years has had to spend a lot of time and energy on what is essentially a marketing activity, which is why I dubbed it “markelopment” (a deliberate riff on “devops”) on Twitter. I’m not among those who think presentations are always evil and should be shunned, but after having created/delivered quite a few presentations and sat through a great many more, I think I’m in a position to offer just a little bit advice.

First, I’ll say that hundred-slide decks annoy me. Yes, I know it’s usually a reaction to the problem of slides that are too few and too densely packed, leading to the also-awful phenomenon of the presenter spending most of the time just repeating what everyone can already read, but it’s an over-reaction. The other day I was reading some slides online, and I encountered the following pattern:

Slide N-1: (clip art)
Slide N: “vs.”
Slide N+1: (more clip art)

A whole slide just for “vs.”? That’s wasting my time. Presenters who use that style end up spending too much of their presentation actually changing slides and waiting the obligatory five seconds for the audience to catch up, no matter how little content is on each. Stephen Foskett pointed out that Lawrence Lessig only puts one word on each slide and is still a very highly regarded speaker. Well, yeah, he’s Lawrence Lessig. I’m not, you’re not, and probably neither is anyone you know (unless of course you know Lessig).

Now, I know presentation length can be tricky. I myself do tend to err on the side of making my slides too busy and very spare graphically. I do that because I know that the slides are likely to be viewed more in email etc. than with me actually presenting them, so to make sure they’re useful as a reference I often sacrifice a little on the “live” side. What I’d generally like to do is create two decks – one verbally spare and graphically rich to illustrate or anchor what I’m saying live, and a longer form for sending around later. That means even more time spent in Impress, though, and is often not feasible for various other reasons as well. My best advice is to determine a good “minutes per slide” figure based on the content, the audience, and an honest appraisal of your own ability to keep the audience interested while the slides aren’t changing, then use that to determine an appropriate slide count. If you’re a very dynamic speaker, you can go the Lessig route and spend five minutes on a one-word slide. If you need a hundred slides to fill a thirty-minute presentation, then maybe you’re admitting something about your speaking skills or the intrinsic value of what you’re presenting.

Second lesson: don’t get too cute. I’ve seen too many presentations lately, especially in the “edgier” tech areas, where the author had obviously spent way more time on finding funny clip art and quotes than on the actual content. Again, it’s a balancing act. Humor is good. A good quote or graphic can be an absolutely fantastic anchor for an important point, which you then elaborate or build on verbally. One not-really-funny slide after another after another with too little in between is just distracting.

Another error that I find even less excusable is simple ugliness. Yesterday I saw a presentation which had been done entirely – from title to closing – in what looked like a version of Comic Sans done to look like paint-brush strokes (house painting, not portrait painting). It wasn’t very readable, and looked totally amateurish. I was embarrassed for the author.

Now, somebody’s probably going to think I’m saying that I’ll totally dismiss an otherwise good presentation of an important idea because of slide count or graphics or font choice. Not so. I’ll still listen, but it will cost the author a “point” in my mind. It’s worth keeping in mind that, in these situations, every single point can matter. If you’re presenting to hundreds of people and only care if one or two respond in any significant way to what you’re saying, then maybe none of this matters. Far more presentations are given in smaller groups, though, where the opinion of everyone at the table does matter. People being how they are, they will use all sorts of nuances to form an impression of whether you’re smart, whether you’re trustworthy, etc. It probably won’t be one big thing that causes you not to get that next meeting, but an accumulation of little things. (If you think “meritocratic” open-source techies are any different, BTW, you’re just kidding yourself. The standards are different, but they’re just as stringently applied. Set the wrong tone and you’ll be written off just as surely and completely.) Why give someone the chance to think that you’re too serious or too frivolous, that your presentation shows disorganization, poor prioritization or disrespect for others’ time or sensibilities? Focus on content, by all means, but take just a little time to make sure it’s being delivered in a way that will ensure a good reception.

Everyone knows that the market for products can be very competitive, and that the competition can often take on a distinctly shady character. What seems to be less appreciated is that the “marketplace of ideas” can be just as competitive, and often just as shady. The battles for “mind share” between one project and another, or even between one approach and another, can be fierce. At the more obviously commercial end of this spectrum are debates like NAS vs. SAN, 10GbE vs. IB, iSCSI vs. FCOE, end-to-end vs. embedded functionality. Often the commercial interests and biases of the participants are obvious, but other times less so. When Joe Random Blogger weighs in on one of these debates, finding out that Joe spent months and thousands of dollars to become a CCIE might shed some light on their vested interest. Other times, there’s no such obvious marker but the vested interest is just as real. This is why Stephen Foskett and others have called for explicit disclosure of such interests by bloggers. Full disclosure: I’ve met Stephen, I like Stephen, we’ve done an Infosmack podcast together and he invited me to drop by at a Tech Field Day where I got to enjoy some free food at EMC’s expense. See how easy that was?

Where things get a little murkier is where there’s not an obvious product involved, or where some of the projects are non-commercial. Some of the participants in the fierce SQL vs. NoSQL debates have a commercial stake, some have a personal stake, and some really have no stake at all beyond a desire to see open discussion advance the state of the art. Quick: which category do I fit into? I’m not even sure myself. I like to think I fit into the last category, but this stuff is not entirely unrelated to the job that puts food on my family’s table so arguments that I belong in one of the other categories have plenty of merit. It’s funny that in my day job I rarely get exposed to such conflicts of interest but as a blogger I do. I’ve been asked to write “objectively” about certain products or projects, or sometimes to refrain from writing about them. I’ve tried to ignore those requests as much as possible, and just write about what interests me . . . but I digress. What are we to make of sniping e.g. between Cassandra and HBase advocates, for example, when both are open source? At an even more abstract level, what about centralized metadata vs. “floating master” vs. peer-to-peer distribution, or using the same vs. separate algorithms for wide-area and local replication? What about public cloud vs. private cloud and the people who claim one or another is beneath contempt? The resolutions of such debates have clear implications for certain projects, and the participants are hardly unaware of that, but the debates are not directly about the projects.

All of this creates an environment ripe for manipulation by the less ethical. Here on this blog I’ve often posted about apparent instances of FUD and astroturf, which are two forms of such manipulation. Bloggers or Twitterers with undisclosed interests in one side of the debate are everywhere. I’ve seen one fellow with a vested interest in certain NoSQL projects repeatedly bash other projects quite savagely, more than once, for faults that his own pet projects still have or had until only a week before, without disclosing his direct involvement in the alternatives that remain after the bashing is done. One of the dark sides of open source is the practice of mining competitors’ code for flaws not so they can be fixed but so that they can be used as ammunition in the war of ideas. Perhaps the most effective technique I’m aware of in this area, though, is the wooing of converts. As much as we all like to pride ourselves on being completely rational and empirical, technology is still a social enterprise and nothing can help one side of a debate more than winning over a prominent member of the other team. “I used to believe in SAN/embedded/centralized but I’ve seen the error of my ways” can be very powerful. It strongly implies that the evolution from novice to expert and that from one position to the other are somehow linked. Novices are fooled into believing X, but experts have figured out Y. Sometimes I’m sure the change in heart is legitimate and sincere, driven by increased knowledge just as it seems or perhaps by the ever-changing tradeoffs we all have to make. (I’ve done this myself, with regard to storage vs. processing networks and the balance of traffic between the two in a distributed filesystem.) Other times, I’m just as sure that someone’s change of heart is the result of deliberate persuasion. Contact might have been deliberately made, perhaps through a mutual friend. The strongest features, situations, and (perhaps non-obvious) future plans/directions for one alternative might have been shared, all deliberately planned but presented as innocent exchange of ideas between colleagues. Sometimes a convert can be won this way, and since nobody is ever as zealous as a new convert the result can often be advocacy of the new preference even in situations where the old preference remains objectively better. Yes, you can buy publicity like that. What you can also do is create “moles” who gain some level of notoriety within a community – it’s really not that hard with emerging technologies – and then very noisily “defect” to an opposing camp.

I don’t know for sure how prevalent this sort of manipulation is. I’ve certainly seen plenty of astroturf and FUD, I’ve seen some of the attempts to persuade “thought leaders” one way or another, but I don’t know for sure if I’ve ever actually seen a mole. What I do know is that I don’t see everything, and I’d be a fool to believe these things don’t happen. The means, motive, and opportunity are all there. There are “social media experts” who are paid – and paid quite well – to do almost exactly what I’ve described; I’m sure not all of them have 100% clean hands. I’ve yet to meet a VP of marketing at a startup who would have any qualms, who would hesitate one second, over paying someone to do these things if they thought that person had the capability. I’m not saying we should all start jumping at shadows, but if you see a prominent advocate of low-cost open-source scale-out solutions suddenly start singing the praises of a vendor who is notoriously opposed to all three features, maybe you should at least consider the possibility that their change of position is something other than a total accident.

The first rule of self-help scam artists is to tell people what they want to hear, without being too obvious about it. Tell them that there’s an easy way for them to get whatever they want, or that everything bad is somebody else’s fault, but wrap it up in convincing language that makes it seem somehow logical. One example of this that has really been driving me nuts lately is Seth Godin’s “linchpin” idea. Apparently the idea is that everyone should aspire to be a post-industrial “artist” instead of just a cog in a machine, everyone has the potential to do “emotional work” at a “high level” and thus become indispensable, etc. It’s a very positive message, but I’m just not buying it.

First, while I believe it’s not possible to predict who will rise to the challenge of becoming a linchpin, and thus that we should give everyone the chance, that doesn’t mean absolutely everyone can. Many people really are cogs in the corporate machine, are not particularly capable of being anything else, and are even happy that way – saving their creative energy for other pursuits such as family, hobbies, sports, and so on. Ninety percent of the people who proudly portray themselves as linchpins quite notably do not meet the criteria, and I’d even say that they’re less likely than average to be true linchpins because the significant time they spend on self-affirmation and self-promotion is time not spent actually doing anything that would make them real linchpins. My second objection to Godin’s idea is that just because some people can be linchpins doesn’t mean all can. It’s like every child being above average. Show me an organization where every single contributor is indispensable, and I’ll show you an organization that is guaranteed to fail as the normal course of events makes any one of them unavailable. Like it or not, only a few people in any group can be linchpins and if you want to be one then you’ll be in competition with others to find that niche.

We really need to get over the idea that every worker should be a unique and pretty snowflake to deserve a place in a high-functioning organization. Somebody has to do the things that anybody can do, and as long as that’s the case there’s a need for discipline as well as self-expression. If everybody thinks they’re leading, nobody really is. Most people are unique and special in some way, but usually not in a way that any employer/client can or should care about. Ordinary people, or people who are special in non-work-related ways, can still play a valuable and even essential role in even the most creative and innovative environments. Uniqueness is not a requirement. A beautiful snowstorm would still be beautiful even if every single snowflake looked exactly alike. Real self-help would mean teaching people how to find and function in and enjoy being in a creative environment, not just telling them that they can be among its leaders. For many of them, it’s simply not true.

Today I got drawn into a discussion on Sutter’s Mill about standards. The topic at hand was the ISO C++ standard and whether it qualifies as open, but it gave me a chance to read and think a bit more about other issues as well. Let’s start with a little background on how ISO works. Here are some of the more significant details.

ISO standards are developed by technical committees, (subcommittees or project committees) comprising experts from the industrial, technical and business sectors which have asked for the standards, and which subsequently put them to use. These experts may be joined by representatives of government agencies, testing laboratories, consumer associations, non-governmental organizations and academic circles.

Proposals to establish new technical committees are submitted to all ISO national member bodies
…
Experts participate as national delegations, chosen by the ISO national member body for the country concerned

In other words, there is no true individual involvement in the ISO standards process. A standard is initiated by a national standards body – such as ANSI in the US – which also decides who can be a delegate. Oh yeah, and they pay for it too. According to ISO in figures, member bodies pay 140 million Swiss francs per year to run the committees that do the hands-on work of developing standards, vs. 35M CHF for the Central Secretariat itself. 55% of that 35M – not of the entire 175M – also comes from the member organizations. The other 45%, or a mere 9% of the total, comes from publications and other services. This is why I said, in Herb’s thread, that the claim of ISO needing to charge $30 for a copy of the C++ standard is utter crap. They don’t need that fee to support their operations any more than IETF – which publishes their standards completely for free – does. They charge that fee for completely other reasons. This brings us back to the process.

ISO refers to theirs as an open process, and ANSI follows suit, but in fact it’s a very political process. You can’t walk in as an individual, even the most knowledgeable invididual in the world regarding a particular subject, and participate. You have to know or work for someone, who knows someone, and so on. Corporations feed people and proposals to industry or trade associations, which feed them into the national standards bodies, which feed them into ISO. There’s also politics at a higher level, as proposed ISO standards are often national standards somewhere already. Sometimes there are multiple competing national standards involved. Sometimes there are multiple technical bodies within ISO, or other bodies outside ISO (such ITU) wrangling over who controls a standard. In short, it’s a mess. In many cases, users’ needs would be just as well satisfied by allowing standards to remain within their originating national body (as with ANSI C) or industry/trade association (as with Java). Often the only people driving demand for an ISO standard are the very people who get paid to develop the ISO standard, or who hope to profit from it. What a racket.

This brings us back to the question of whether ISO standards are open. Participation is political, because of the strategic value standards have to the various parties at all levels, and thus in effect limited even though it’s nominally open. The output is only available for a fee that is totally unjustified by the cost structure that’s involved, and then subject to copyright. “Open” is the wrong word for that. On the other hand, the fact that the standards are published at all is an improvement over proprietary “de facto standards” and the freedom from licensing fees is an improvement over that. “Available” and “unencumbered” cover those properties, but “open” to me still means something more like the IETF model. A standard is only open if participation does not require sponsorship, and if its text is available for only nominal cost.

For quite a while, my favorite source editor/browser has been KScope. It combines one of the better source editors I’ve seen (Kate) with a very robust cross-referencing tool (Cscope). Unfortunately, when KDE went from version 3 to version 4, they made changes that broke KScope. What’s a little odd is that Kate still works. You’d think that if they’d fixed Kate and maintained its interface, KScope would be unaffected. Apparently, though, whoever made the KDE4 changes to Kate decided to extend the breakage instead of containing it. Nice call, whoever you are. Thanks.

This wouldn’t be worth a rant if I could get KScope running again in reasonable time, but that’s not the case. The KScope developers’ response to the breakage was to push out one last 1.9 release, containing only the features that still worked and removing everything that was useful about the project to begin with, and then stop development. That doesn’t seem like the best response to me, but perhaps they no longer had the time/inclination to deal with this level of breakage. I haven’t yet figured out how to put together enough of a KDE3 environment in Fedora 12 to build KScope 1.6 myself, though this is supposedly possible and I might yet find the magic formula. The lack-of-resources argument doesn’t work quite as well for Fedora as for KScope, but they might also lack volunteers in this particular area. I’m not going to blame them for the way KDE stabbed everyone in the back.

I’ve historically been a fan of KDE. I’ve been using KDE versions of natively-GNOME distributions for quite a while now, because I think it’s better both technologically and at a user-experience level. They totally botched the KDE4 transition, though. As far as I can tell, furthermore, there were plenty of people predicting this disaster. They could have listened, but they did what they wanted to anyway out of sheer arrogance and insensitivity to either users’ or external developers’ needs. I’m simply not a KDE fan any more. If I want to get away from GNOME, I can use XFCE. I might still use some specific KDE programs, but I don’t trust them to provide a complete environment any more.

So, what are my options? I used Source Navigator quite happily for many years, but it now seems crufty and unpleasant. I’ve been using CodeLite a bit because it has tolerable Cscope integration so I can navigate large bodies of code that I’m not intimately familiar with. I’m not a big fan of the Studio/Eclipse workspace/project/virtual-folder model where the IDE tries to write your own (warthog-butt-ugly) makefiles, though, and the editor’s not the best either. Using CodeLite from home is also barely tolerable, whether it’s the program or the files that are remote. Somebody seems unaware that redraws and/or filesystem operations might be expensive enough to expose lazy coding. Anjuta seems like a roughly-equivalent offering, which might improve in some areas. I don’t want to use Eclipse CDT because it’s bloated or KDevelop because it’s KDE, but I might be forced to use one or the other if Anjuta doesn’t pan out. Sigh. I suppose I could fix KScope if I didn’t mind giving up all of my remaining free time for the next long while, but I wish such choices weren’t made necessary by others’ bad behavior.

First, a lot of people have probably heard that Microsoft lost an appeal in a $290M case against Canadian firm i4i.

A federal appeals court on Tuesday affirmed a $290 million patent infringement judgment against Microsoft Corp. (MSFT) and reinstated an injunction that bars the company from selling current versions of its flagship Word software.

The U.S. Court of Appeals for the Federal Circuit said the injunction will go into effect Jan. 11.
…
After the jury verdict, U.S. District Court Judge Leonard Davis issued a permanent injunction that barred Microsoft from selling Word 2003 and Word 2007, which use a technology called “Custom XML” that is used to classify documents for retrieval by computers. Davis also ordered Microsoft to pay more than $290 million in penalties.

Barred from selling one of their flagship products? Wow. I managed to glean some idea about custom XML is, but I still can’t quite figure out what it does in the sense of providing any kind of useful or significant functionality to end users. It seems like one of those things that gets argued about in standards meetings, mostly by people who are more confident of their ability to keep up with developments in such intellectually inbred communities than in the wider world of real-life computing, and nobody cares until it hits the patent system. An interesting sidelight is that the original judgement seems due in part the result of a conflict between one of Microsoft’s lawyers and the East Texas judge. That’s a fight where it’s really hard to root for either side, but the practical consequence is that it might give Microsoft grounds for further appeal. By the way, how does a company from Toronto get to sue one from Redmond in a Texas court? That’s not only the wrong state but the wrong country.

So the news about Microsoft and i4i might be a mixed bag, but the other piece of news is quite a bit better. The US Board of Patent Appeals and Interferences (the what?) issued a ruling about the patentability of machines using mathematical algorithms. The ruling is from August, but it just became “precedential” this past Monday. I have no idea what that means, except maybe that those who find XML standards too accessible should consider a career in patent law. Anyway, here’s the good part.

The BPAI’s test for a claimed machine (or article of manufacture) involving a mathematical algorithm asks two questions. If the a claim fails either part of the two-prong inquiry, then the claim is unpatentable as not directed to patent eligible subject matter.

(1) Is the claim limited to a tangible practical application, in which the mathematical algorithm is applied, that results in a real-world use (e.g., “not a mere field-of-use label having no significance”)?

(2) Is the claim limited so as to not encompass substantially all practical applications of the mathematical algorithm either “in all fields” of use of the algorithm or even in “only one field”

In a nutshell, this reaffirms the principle that only the application of an algorithm – not the algorithm itself – can be patentable. This has supposedly been true all along, but many patents on applications have effectively become patents on algorithms because the applications have been defined so vaguely or broadly that no other application can escape claims of infringement. This ruling says that failure to limit the scope of such a patent properly might not only cause a specific infringement case to be dismissed but might also cause the patent in its entirety to be invalidated. This could fundamentally change the way people pursue such patents. Until now the incentive has been to make claims as broad and vague as possible, because there was no risk in doing so. Now, though, there’s a risk that over-reaching might destroy the patent’s value entirely, so there’s an incentive to be more specific. Only time will tell whether it actually works out that way, but it’s a good sign.

Now I’m kind of curious about which of my own (five) patents would survive this test. I doubt that they’ll ever be tested, and I won’t provoke my former employers’ legal departments by speculating, but it’s an interesting question.

I wrote about the demise of SiCortex when it happened, but the subject has surfaced again recently – first in a Network World article, and then on Google’s Cloud Computing group. I’d like to address a few issues these raise, plus some that have come up elsewhere. Please bear in mind that these are all from my perspective as an engineer in SiCortex’s software group, so more informed parties might be able to correct some of the details, but I’m pretty confident about the gist of them.

• SiCortex was explicitly not trying to sell a supercomputer. We never even cracked the Top500, or intended to except as a prerequisite for the more relevant Green500 (at the time; the requirements have since changed). We even offered the 12-node 72-core $15K SC072 as an entry point, which was lauded for its affordability and accessibility (and predated Cray’s CX-1 by quite a bit). That’s hardly “Formula One” territory.
• The “green” angle was not just a matter of market positioning, though it was certainly that. Low power and heat were also quite fundamental to achieving the level of density and simplicity (e.g. no active network components other than the compute nodes) that were the real core of the architecture.
• Using an x86 instruction set was not an option. Even if the appropriate licenses had been available, they would surely have been out of our financial reach. As it was, IP licensing (especially MIPS) was the second largest expense behind payroll.
• Similarly, using QPI or similar wasn’t an option. QPI itself didn’t even exist at the outset, and licensing it later would have been similarly expensive.
• Both the x86 and QPI mirages are also affected by the question of technological fit with other key architectural elements (see above). It seems vaguely possible that something could have been worked out with VIA or some other third party that already had the necessary licenses, but then again maybe not. The same question came up with respect to Raza and Cavium in the MIPS space. I was personally never very satisfied with the answers I got when I or others asked about such things, but then I was barely qualified to be asking. Maybe one of my erstwhile colleagues could shed more light here.
• It was probably not feasible to make and sell machines even smaller than the SC072. I and others did favor this idea, largely based on the four-node “Frost” machines we’d used during development, but the people making such decisions said that we could not afford the sales and support infrastructure necessary for that market. Having seen the margins and support costs for the SC072 up close and personal, I agree that it would not have been profitable. An argument might still be made that such a machine would have been worthwhile even if it had not been profitable in and of itself, as a way of establishing mind share etc., but that’s not clearly the case either.

As I said in my other post, the failure of SiCortex had nothing to do with the technical merit of the product. I don’t think they even had to do with the market merit. Don’t believe the disgruntled ex-employee who now works for another player in this space and seems to make a point of going around to every article or story about SiCortex to comment anonymously about CPU or memory performance. Sour grapes, dude. The machine was selling OK, though of course not as well as anyone had hoped. Where we failed was in “selling” the company to investors so we could finish building the next generation, and in any other VC market even that might have been an easy sell. After all of the times I’ve seen charlatans do a better job of selling a story to investors than of selling a product to users, I think SiCortex still stands as an example of the exact opposite. We were for real. It’s the investors who turned out to be fake.

Saying “X is dead” is the new way to put on jaded-veteran airs so you can look down at all those people who don’t know it’s dead yet. Information technology is dead. Operating systems are dead. Feh. Let’s take a look at the “IT is dead” claim, as recently made by Thomas M. Siebel in the NYT, first.

In Mr. Siebel’s view, I.T. is a mature industry that will grow no faster than the larger economy. He contends that its glory days are past — long past, having ended in 2000.

I have no doubt that IT as Siebel knows it is starting to decline, but that only reflects how much IT has continued to evolve while he has apparently stood still. Siebel is a typewriter manufacturer. Back in the day, typing was something that typists and secretaries did. When computers came along, everyone started typing. I’m sure a typewriter of that time might have looked at the declining number of typists and thought “typing is dead” but that would have been a mistake. Don’t confuse the decline of a market with the decline of an underlying technology or activity. In fact, more people are typing now than ever. There’s even a lot of money to be made selling keyboards, though they’re computer keyboards instead of typewriters.

So it is with IT. People are still doing IT, but more people are doing it in more places. You no longer need dedicated equipment or personnel to handle even the simplest computing tasks. Users themselves are installing software, updating web pages, etc. Then there’s that person in every office who does slightly less routine kinds of IT tasks like troubleshooting network problems. Routers and even low-end storage are now sold as office supplies, not as specialized computing equipment. Your DVD player has processors running software, but I’ll bet it doesn’t show up on “IT equipment” sales. My iPod Touch has more computing power than things that used to be called supercomputers, but it’s not “IT equipment” either. A lot of this spending doesn’t show up as IT spending, so people like Siebel don’t figure it into their calculations, but that’s just because they’re counting the wrong thing.

There’s a similar dynamic at work with the “operating systems are dead” meme, which has actually been going around for a while. I’m going to pick an example from Paul Maritz, as quoted by Storagezilla.

Paul Maritz pointed out that today no one really writes to an OS anymore, they write to 3rd generation Frameworks. And he’s right, if we look around we see lots of 3rd generation Frameworks and some of them are more cross platform than others. We all know about Java and .Net but Cocoa for MacOS X (Found on your Mac and your iPhone) Adobe Air, Microsoft Silverlight and so on are all examples of Third Generation Frameworks.

Yes, there are a lot of people working on such frameworks, and even more using them. That’s nothing like “nobody writes to an OS any more” though. Who wants to run any of the following on Air or Silverlight?

• Development tools, starting with compilers and interpreters.
• Databases.
• Web servers (on which many frameworks themselves depend).
• Email servers.
• LDAP servers, VPN gateways, etc.
• Anything in HPC.

The answer is nobody. I don’t think the people behind gcc or Oracle or Exchange are planning to run on top of any such “third generation” frameworks (and BTW I wonder how the third generation is different). They plan to support such frameworks, but that’s different. Operating systems provide a general and stable interface, suitable for many kinds of applications beyond those for which any given framework is likely to be appropriate, and with full performance for when that matters

What might be true is that, the division between “back end” platform (i.e. OS) specialists and “front end” interface or domain-specific specialists will change. Instead of every application-development group being about evenly split between the two, there might be some people developing a few (or a few dozen) frameworks and a much larger number using those frameworks. Same as it ever was. Having been an OS developer for so long, I’m well aware that my kind have long been outnumbered by people working at a much higher level of abstraction who barely understand what I do. Maybe framework developers feel the same way, or soon will. That doesn’t mean OSes are dead, though. People still need device drivers and filesystems and network stacks and schedulers and virtual memory and all that other stuff. OS development will continue, and OSes will continue to compete to capture developers’ hearts and minds. A significant number of people will be watching, and using the results directly to improve old frameworks or create new ones. In fact, the prevalence of framework-oriented workloads is causing a near-term increase in the amount of work that OS folks must do to accommodate those workloads. That doesn’t seem very dead to me.

IT is not dead. OSes are not dead. They’ve evolving. Those who get left behind might find themselves alone and think the party’s over, but in reality they just missed it.