The World Is Not Flat

Way back when I was a young pup, either in college or after that but before I started my career, I got to use an operating system called MTS. That stands for Michigan Terminal System. It was created to run on IBM (and later Amdahl) mainframes, when U of M got tired of waiting for IBM to deliver a multi-user operating system. Like most code that old, it was an interesting combination of ideas that have since been abandoned because they were stupid, ideas that were ahead of their time, and ideas that were somewhere in between. Here are some of the more interesting ideas.

  • The filesystem had a feature to include the entire contents of one file at a specific point within another. Who needs symbolic links when you can just create a file containing a single %include directive? Why would programming languages have to synthesize this behavior in a bazillion subtly different ways if the basic functionality existed natively in the OS? Yeah, I know, record-oriented filesystems (basically a prerequisite for this) lost out to simple byte-streams for many good reasons, but every victory comes at a cost.

  • MTS had a very robust ACL system, which allowed you to control access by user, group, or "pkey" (i.e. what program was running). Much better than set-uid in my opinion.

While I was still using MTS, they added a macro system - what we would now think of as a shell scripting language. One of the very first uses of this macro system was to sythesize a hierarchical directory structure on top of the flat one native to MTS. I really wish I could remember the name of the author, to give credit. He was a Computing Center consultant, and this would have been in 1985 or so, if anybody wants to help me out. It was a pretty slick combination of naming conventions and macros, and I think it made many users' lives easier.

The reason I started thinking about MTS is that I see people doing the exact same things now - nearly thirty years later - to simulate a hierarchical namespace on top of the flat one provided by most object stores. Let me repeat something I've said many times before, in many ways: flat namespaces weren't just crap in DOS, they were crap in MTS even before that and they're still crap today. Crap, I say. Anybody who implements a supposedly modern file/object store with a flat namespace is simply screwing their users to suit their own convenience. The scalability arguments don't hold water, because the scalability issues mostly have to do with the operations that you have to support (e.g. atomic rename) than with whether or not you have nested directories. This is something that has to be built into the data store, with the necessary recursive name resolution done one place one time by people who understand that data store, instead of being done ten incompatible ways by ten different outsiders. Even quite smart people can trip when they try to bolt on a hierarchical structure after the fact.

Users have shown over and over again that they want flexibility to organize and reorganize their data, in ways richer than a flat or even single-level hierarchy will allow. Maybe there's an even better way, but so far none of the attempts to replace nested directories with tags or links or database-like queries seem to have gained much traction. Until someone comes up with something better, the nested-directory structure should be considered a minimum standard for anything that's supposed to replace the traditional filesystem.

Comments for this blog entry