My MacBook Pro Adventure

In the little more than a year since I got it, my MacBook Pro’s battery life has gone from 3-4 hours to barely 1.5 hours. That’s disappointing. Apparently the software that figures out how much time the machine has left never caught up to this reality, because in the last few weeks I’ve had problems with the machine suddenly losing power when it supposedly had half an hour or more left. That’s aggravating. On Friday, after one of these episodes, the machine couldn’t boot and the Apple-provided repair utility (run from DVD) was powerless to help. That’s disgusting. Something like that is not just a technical error. It’s also a betrayal of trust because, if Apple had followed well established standards for how filesystems should be developed and tested, this kind of failure should never occur on functioning hardware – and my hardware, by the way, is fine.

Before I go on, I’d like to say that things seem to be OK. Thanks largely to Data Rescue, I was able to slurp all of the data I cared about from the disk. After reinstalling, things seem reasonably OK, though I’m sure I’ll be reinstalling various bits of software for a while yet. The net result is a loss of convenience and confidence, but not – as far as I’ve been able to tell so far – of data. It’s particularly notable that iPhoto seamlessly dealt with my reimportation of all of my pictures. Kudos to those developers for that. Now, on with the rant.

The thing that infuriates me about this, as a professional filesystem developer, is how contrary it is to how I and my colleagues generally work. Let’s leave aside the fact that I was playing Zuma at the time and had practically nothing going on in the background, so there was no reason anything should have been touching volume-level data structures on disk for at least the last several minutes. Even if there had been a reason for such activity, the whole point of a journaling filesystem is supposed to be that updates are done in such a way that this kind of catastrophic loss of access to data cannot persist beyond the next journal replay. Recovery after sudden power loss is something that Apple should be testing all the time. Considering the amount of time they’ve had to discover and fix any bugs in this area, such conditions should be all but impossible for users to trigger – and yet it doesn’t take long for Google to tell me that I’m far from alone on this one. There’s even a considerable market for software to perform data recovery and repair that Apple’s own utilities cannot. With all due respect to the fine folks at Prosoft Engineering and elsewhere, this represents yet another abject failure by Apple. Their crappy battery, their crappy battery-monitoring software, their unbelievably crappy filesystem, and their crappy repair utility all contributed to this.

Every developer, tester, or project/product manager at Apple who ever had anything to do with HFS(+) should be ashamed.

Is POSIX Still Relevant?

When I talk about working on a cloud filesystem, I often describe its interface as POSIX or near-POSIX. Most people just accept that, a few kind of look blankly, and a few use it as a cue to ask, “Is POSIX still relevant?” Unfortunately, when this occurs I’m usually in an environment where I have to suppress the eye-roll this induces. A little knowledge is dangerous, and people who ask that question are the living proof. “POSIX” in this context basically just means a familiar filesystem, as implemented by any sane VFS layer (plus Linux’s), with approximately the following functionality.

  • Hierarchical directory structure (readdir, creat/mkdir, link/symlink)
  • Inode abstraction with attributes such as owner, permissions, times (stat/fstat, chown/chgrp/chmod)
  • Read and write of simple byte streams (open/close, read/write, fsync, ftruncate)

Your email is almost certainly stored in a POSIX filesystem somewhere, as are all of your spreadsheets and photos and source code etc. Most of what web servers deliver comes from a POSIX filesystem either directly or indirectly (e.g. from an RDBMS which is loaded from a filesystem and stores data in a filesystem). For any one of these examples there might be other options as well, but not always (I’ve yet to see a compiler that operates on any other kind of storage) and in any case there’s only one that satisfies all of these needs – all day, every day, for billions of users. The people who question POSIX’s relevance generally don’t know how their supposed alternatives actually work, or that in the dim and distant past there really were alternatives to the POSIX model but those alternatives were found sorely lacking. (The same, BTW, is often true of other POSIX standards besides those involving filesystems.)

All that said, most filesystems are only 99% POSIX. Many of the divergences have to do with consistency/ordering/durability issues. NFS is (in)famous for cutting corners in many of these areas, as are ext[234], and most people neither know nor care until they try to run their software on a different system and it unexpectedly fails (e.g. because PVFS2 doesn’t allow access to unlinked files). For the most part, though, relatively trivial differences such as character sets and name lengths and forward-slash vs. back-slash are the only ones people notice. So go ahead and use whatever other storage model you want. Just remember that sooner or later your data will probably pass through a POSIX filesystem (and from there to a T10/T11/T13 block device) which has a lot to say about how it will perform or how well it’s protected. Too often I’ve seen people test some other kind of storage that runs on top of a filesystem, just assuming that the filesystem will do its job, then use those very same results to proclaim the irrelevance of POSIX. It’s like driving to a conference where you proclaim the irrelevance of cars.

Open Source vs. Open Community

Today I had a great email discussion with Artur Bergman about the work he’s been doing on riakfuse. We compared notes on dealing with rename, writing a FUSE driver in C/C++ vs. Perl/Python, etc. It was most enjoyable, and I said so on Twitter. I also said that “open communities rock” and made a comment about open communities vs. open source. That piqued a few people’s interest, so I’ll try to explain.

Open source means that somebody can get the source code for something without having to pay a license fee or sign an NDA, and may pass it on to others likewise without getting into copyright trouble. There are finer distinctions to be made between various kinds of open source and free software, but that’s close enough for current purposes. An open community, by contrast is one in which “outsiders” are able to ask questions and contribute ideas, possibly even contribute code, without having to deal with unnecessary barriers. Both are departures from the (now slightly dated) corporate norm where only employees have any access whatsoever to either the code or the process of its development.

It’s important to note that the two forms of openness almost always go together. The vast majority of open-source projects also represent open communities. However, there are places where the connection breaks down. Sometimes code is available, but the development process is utterly closed. This is clearly the case with code that’s released as open source by a company (and often abandoned soon afterward but that’s another topic). There are also many personal or small-group projects where the code is released without even an intent of building a community around it, and of course complaints about this or that project being too cliquish/elitist to accept outside contributions are legion. I’m not talking about necessary limitations on who can modify the code, or about individual developers being jerks. One could hardly work at Red Hat and be unaware of either phenomenon. I’m talking about institutionalized hostility toward “outsiders” from top to bottom, with no reason beyond a desire to keep the circle small. On the flip side, many academic research projects never release code or even have code as an intended artifact, but nonetheless are entirely open and willing to engage in dialog about technical details. Some standards groups such as IETF also operate on such a basis at least some of the time.

In many cases, an open community is more important than open source. Why? Because code is often useless. Even as a coder myself, I’d often prefer a short textual description of an algorithm or configuration detail to a brusque “read the f-ing code” from the developers. Sure, you can build from the code, but if it’s written in a language you can’t reasonably be expected to know or frameworks and conventions that are themselves completely unspecified then you can’t do much of anything else with it. In particular its presence doesn’t further the exchange of ideas and improvement in overall state of the art. I’ve had to delve into such code hundreds of times, even to the extent of what most would call reverse engineering, but the exercise is more akin to criminal forensics than to an open exchange of ideas . . . and that’s likely to be even more true for someone less fluent in code than I am. To the extent that the brief note is more useful for some purposes than the complete code, it can also be a better indicator of true openness.

Ideally, of course, both code and community should be open. Again, this is already the case most of the time. I’m not trying to make any grand philosophical point here, but I do think it’s important to remember that the two kinds of openness are not necessarily coincident. If openness is a goal, then both forms have to be considered separately and not squashed together as though they were one.

Stateless = Clueless

I found this gem on imap-uw (via @hakmem) today.

The authors of Outlook and Thunderbird are victims
of a computer science course mindset which, starting in the 1980s, taught
their pupils that all protocols are (or should be) stateless. Thus, they
believe that IMAP is like HTTP; that when a server fails to respond
immediately, that means that the correct remedial action is to disconnect
and try again, or just disconnect and assume that everything happened
anyway.

I’ve heard many of the arguments for “stateless” protocols before, mostly before the current crop of advocates even entered the computing field (but more about them later). Too often, though, “stateless” just means that the state is harder to find. Usually it’s still there in the form of server-side session objects, client-side form fields, cookies, transaction IDs, or other devices. Sometimes it’s even more insidious, hidden in a configuration of the underlying data that implicitly limits what requests can still succeed. I’ve written about the evils of such implicit state before. When stateless-protocol dogma forces you to throw away context and then recover it later so that you can validate a new incoming request, the result is at best inefficient and quite often unreliable. Separating context from connection might be a good idea for various reasons, but it doesn’t really make the need for state go away. The only way there can truly be no state is if there are no rules whatsoever for when any data might be overwritten by any requester, and that’s just not a useful paradigm in the vast majority of cases.

This is related to why I dislike the currently fashionable REST dogma. REpresentational State Transfer is inherently a shared-state programming model. I’ve often said that sharing state without thinking about consistency is insane, and I’d say the same about sharing state without thinking about concurrency issues (e.g. races and deadlock) as well. It’s fine if you decide that you don’t need consistency or locking, so long as it’s a deliberate and considered choice which the rest of your application (or user base) can deal with. What’s not fine is just assuming that whatever functionality you happen to have in those areas is what you need, and yet that’s exactly what REST encourages people to do when the functionality is that of HTTP. Here, Mr. Client, I’ll give you a slice of my state, I might give overlapping slices to a bunch of other clients as well, any of you can come back with arbitrary changes at any time and I promise I won’t maintain any of the state necessary to validate those changes against one another (though I will do authentication because that’s the one lecture my authors didn’t sleep through). Sheer lunacy. At least Roy Fielding seemed/seems aware that his analysis of REST vs. alternatives was specific to a particular kind of application at a particular point in time, but many RESTafarians take a far less nuanced position. What’s truly amazing is when people with a thorough grounding in better concurrent/distributed programming models, like the actor model, promote the antithetical REST for common use. Really, sometimes REST is the right choice. Sometimes RPC is the right choice. Sometimes what you really want is server-driven notification requiring more context than either. The idea that all kinds of computer interaction in 2010 should be based on a protocol for serving static content to passive users in 1992 belongs in the dustbin of history.