I hate spending half my time dealing with build systems, source-control systems, package managers, and such. There are too many out there, they all suck, everybody has their favorite one and their favorite way of using it, and they’re not at all shy about ramming their preferences down your throat . . . which brings me to my real point. I hate programmers. Hot damn, but we are a noxious breed, aren’t we? I’m tired of the backstabbing, the trashing each others’ work, the holier-than-thou attitude from the GNU types, the rampant sexism, the bike-shedding, the endless effort to do and re-do all the fun stuff while dumping as much work as possible onto one’s peers, and on and on and on. I know I’ve exemplified many of these sins myself, I don’t need anyone else to tell me that, but if I made it my life’s goal to be as much of a jerk as possible I’d still find myself outdone just about every day by people who aren’t even at their worst.

Of course, I don’t know what else I’d do that pays the bills, so you all are stuck with me, but come on, people. Let’s stop sucking all the fun out of this profession.

One of the most popular approaches to ensuring data protection is immutable and/or append-only files, using ideas that often go back to Seltzer et al’s log structured filesystem paper in 1993. One key justification for that seminal project was the observation that operating-system buffer/page caches absorb most reads, so the access pattern as it hits the filesystem is write-dominated and that’s the case for which the filesystem should be optimized. We’ll get back to that point in a moment. In such a log-oriented approach, writes are handled as simple appends to the latest in a series of logs. Usually, the size of a single log file is capped, and when one log file fills up another is started. When there are enough log files, old ones are combined or retired based on whether they contain updates that are still considered relevant – a process called compaction in several current projects, but also known by other names in other contexts. Reads are handled by searching through the accumulated logs for updates which overlap with what the user requested. Done naively, this could take linear time relative to the number of log entries present, so in practice the read path is often heavily optimized using Bloom filters and other techniques so it can actually be quite efficient. This leads me to a couple of tangential observations about how such solutions are neither as novel nor as complete as some of their more strident champions would have you believe.

• The general outline described above is pretty much exactly what Steven LeBrun and I came up with in 2003/2004, to handle “timeline” data in Revivio’s continuous data protection system. This predates the publication of details about Dynamo in 2007, and therefore all of Dynamo’s currently-popular descendants as well.
• Some people seem to act as though immutable files are always and everywhere superior to update-in-place solutions (including soft updates or COW), apparently unaware that they’re just making the complexity of update-in-place Somebody Else’s Problem. When you’re creating and deleting all those immutable files within a finite pool of mutable disk blocks, somebody else – i.e. the filesystem – has to handle all of the space reclamation/reuse issues for you, and they do so with update-in-place.

Despite those caveats, the log-oriented approach can be totally awesome and designers should generally consider it first especially when lookups are by a single key in a flat namespace. You could theoretically handle multiple keys by creating separate sets of Bloom filters etc. for each key, but that can quickly become unwieldy. It also makes writes less efficient, and – as noted previously – write efficiency is one of the key justifications for this approach in the first place. At some point, or for some situations, a different solution might be called for.

The other common approach to data protection is copy on write or COW (as represented by WAFL, ZFS, or btrfs) or its close cousin soft updates. In these approaches, blocks are updated in place, but with very careful attention paid to where and/or when individual block updates actually hit disk. Most commonly, all blocks are either explicitly or implicitly related as parts of a tree. Updates occur from leaves to root, copying old blocks into newly allocated space and then modifying the new copies. Ultimately all of this new space is spliced into the filesystem with an atomic update at the root – the superblock in a filesystem. It’s contention either at the root or on the way up to it that accounts for much of the complexity in such systems, and for many of the differences between them. The soft-update approach diverges from this model by doing more updates in place instead of into newly allocated space, avoiding the issue of contention at the root but requiring even more careful attention to write ordering. Here are a few more notes.

• When writes are into newly allocated space, and the allocator generally allocates seqential blocks, the at-disk access pattern can be strongly sequential just as with the more explicitly log-oriented approach.
• The COW approach lends itself to very efficient snapshots, because each successive version of the superblock (or equivalent) represents a whole state of the filesystem at some point in time. Garbage collection becomes quite complicated as a result, but the complexity seems well worth it.
• There’s a very important optimization that can be made sometimes when a write is wholly contained within a single already-allocated block. In this case, that one block can simply be updated in place and you can skip a lot of the toward-the-root rigamarole. I should apply this technique to VoldFS. Unfortunately, it doesn’t apply if you have to update mtime or if you’re at a level where “torn writes” (something I forgot to mention in my “how to lose data” post) are a concern.

It’s worth noting also that, especially in a distributed environment, these approaches can be combined. For example, VoldFS itself uses a COW approach but most of the actual or candidate data stores from which it allocates its blocks are themselves more log-oriented. As always it’s horses for courses, and different systems – or even different parts of the same system – might be best served by different approaches. That’s why I thought it was worth describing multiple alternatives and the tradeoffs between them.

• Don’t provide full redundancy at all levels of your system. It’s amazing how many “distributed” systems out there aren’t really distributed at all, leaving users entirely vulnerable to loss or extended unreachability of a single node, without one peep of protest from the people who are so quick to point the finger at systems which can at least survive that most-common failure mode.
• Be careless about non-battery-backed disk caches. If data gets stranded in a disk cache when the power goes out, it’s no different than if it was stranded in memory, and yet many projects do absolutely nothing to detect let alone correct for obvious problems in this area.
• Be careless about data ordering in the kernel. My colleagues who work on local filesystems and pieces of the block-device subsystem in Linux (and others working on other OSes) have done a great deal of too-little-appreciated work to provide the very highest levels of data safety that they can without sacrificing any more performance than necessary. Then folks who preach the virtues of append-only files without knowing anything at all about how they work turn around and subvert all that effort by giving mount-command and fstab-line examples that explicitly put filesystems into async mode, turn off barriers, etc.
• A special case of the previous point is when people actually do seem to know the options that assure data protection, but forego those options for the sake of getting better benchmark numbers. That’s simply dishonest. You can’t claim great performance and great data protection if users can only really get one or the other depending on which options they choose. Pick one, and shut up about the other.
• Be careless about your own data ordering. A single I/O operation can require several block-level updates. Many overlapping operations can create a huge bucket of such updates, conflicting in complex ways and requiring very careful attention to the order in which the updates actually occur. If you screw it up just once, and it takes a special brand of arrogance to believe that could never happen to you, then you corrupt data. If you corrupt metadata, you might well lose the user data it points to. If you corrupt user data that can be even worse than losing it, because there are security implications as well. It’s not nice when some of your confidential data becomes part of somebody else’s file/document/whatever. At least with mmap-based approaches, it’s fairly straightforward to do things with msync and fork and hypervisor/filesystem/LVM snapshots to at least guarantee that the state on disk remains consistent even if it’s not absolutely current.
• Don’t provide any reasonable way to take a backup, which would protect against the nightmare scenario where data is lost not because of a hardware failure but because of a bug or user error that makes your internal redundancy irrelevant.

Of course, some of these issues won’t apply to Your Favorite Data Store, e.g. if it doesn’t have a hierarchical data model or a concept of multiple users. Then again, the list is also incomplete because the real point I’m making is that there are plenty of data-protection pitfalls and plenty of people falling into them. Some of the loudest complainers already had to suspend their FUD campaign to deal with their own data-corruption fiasco. Others are vulnerable to having the same thing happen – I can tell by looking at their designs or code – but those particular chickens haven’t come home to roost yet.

Look, I laughed at the “redundant coffee mug” joke too. It was funny at the time, but that was a while ago. Since then it’s been looking more and more like junior-high-school cliquishness, poking fun at a common target as a way to fit in with the herd. It’s not helping users, it’s not advancing the state of the art, and it’s actively harming the community. As one of the worst offenders once had the gall to tell me, be part of the solution. Find and fix new data-protection issues in whichever projects have them, instead of going on and on about the one everybody already recognizes.

• The HS article says that “Pomegranate should be the first file system that is built over tabular storage” but that’s not really accurate. For one thing, Pomegranate is only partially based on tabular storage for metadata, and relies on another distributed filesystem – Lustre is mentioned several times – for bulk data access. I’d say Ceph is more truly based on tabular storage (RADOS) and it’s far more mature than Pomegranate. I also feel a need to mention my own CassFS and VoldFS, and Artur Bergman’s RiakFuse, as filesystems that are completely based on tabular storage. They’re not fully mature production-ready systems, but they are counterexamples to the original claim.
• One way of looking at Pomegranate is that they’ve essentially replaced the metadata layer from Lustre/PVFS/Ceph/pNFS with their own while continuing to rely on the underlying DFS for data. Perhaps this makes Pomegranate more of a meta-filesystem or filesystem sharding/caching layer than a full filesystem in and of itself, but there’s nothing wrong with that just as there’s nothing wrong with similar sharding/caching layers for databases. Compared to Lustre, this is a significant step forward since Pomegranate’s metadata is fully distributed. Compared to Ceph, though, it’s not so clearly innovative. Ceph already has a distributed metadata layer, based on advanced distribution algorithms to distribute load etc. Pomegranate’s use of ring-based consistent hashing suits my own preference a little better than Ceph’s tree-based approach (CRUSH), but there are many kinds of ring-based hashing and it looks like Pomegranate won’t really catch up to Ceph in this regard until their scheme is tweaked a few times.
• I’m really not wild about the whole “in-memory architecture” thing. If your update didn’t make it to disk because it was at the end of the in-memory queue and hadn’t been flushed yet, that’s no better for reliability than if you just left it in memory for ever (though it does improve capacity) and if you acknowledged the write as complete then you lied to the user. Prompted by some of the hyper-critical and hypocritical comments I’ve seen lately bashing one project for lack of durability, I have another blog post I’m working on about how the critics’ own toys can lose or corrupt data, and how claiming superior durability while using “unsafe” settings for benchmarks is dishonest, so I’ll defer most of that conversation for now. Suffice it to say that if I were to deploy Pomegranate in production one of the first things I’d do would be to force the cache to be properly write-through instead of write-back.
• I can see how the Pomegranate scheme efficiently supports looking up a single file among billions, even in one directory (though the actual efficacy of the approach seems unproven). What’s less clear is how well it handles listing all those files, which is kind of a separate problem similar to range queries in a distributed K/V store. This is something I spent a lot of time pondering for VoldFS, and I’m rather proud of the solution I came up with. I think that solution might be applicable to Pomegranate as well, but need to investigate further. Can Ma, if you read this, I’d love to brainstorm further on this.
• Another thing I wonder about is the scalability of Pomegranate’s approach to complex operations like rename. There’s some mention of a “reliable multisite update service” but without details it’s hard to reason further. This is a very important issue because this is exactly where several efforts to distribute metadata in other projects – notably Lustre – have foundered. It’s a very very hard problem, so if one’s goal is to create something “worthy for [the] file system community” then this would be a great area to explore further.

Some of those points might seem like criticism, but they’re not intended that way – or at least they’re intended as constructive criticism. They’re things I’m curious about, because I know they’re both difficult and under-appreciated by those outside the filesystem community, and they’re questions I couldn’t answer from a cursory examination of the available material. I hope to examine and discuss these issues further, because Pomegranate really does look like an interesting and welcome addition to this space.

• 6125447: Protection domains to provide security in a computer system
• 6192476: Controlling access to a resource
• 5966702: Method and apparatus for pre-processing and packaging class files
• 7426720: System and method for dynamic preloading of classes through memory space cloning of a master runtime system process
• RE38,104: Method and apparatus for resolving data references in generated code
• 6910205: Interpreting functions utilizing a hybrid of virtual and native machine
• 6061520: Method and system for performing static initialization

First thing to remember is that this is a patent suit, not a copyright suit. That means it’s not about “Java” at all. It’s about certain ways of implementing a dynamic runtime, regardless of what name or input language is used. In that context, 5966702 is probably the most specific to Oracle’s actual Java-runtime technology, and that’s all about class files. The others are pretty general ideas, even if the Java runtime was the first embodiment used in the patent descriptions. For purposes of determining infringement, it’s mostly the claims – not the description – that matter. It’s probably quite premature for anybody who hasn’t looked at the Dalvik code to say whether it infringes most of these patents or not, or whether Google could avoid infringing on these claims without fundamentally changing how Dalvik works.

1. Memcached has no security of its own.
2. Many people deploy memcached to be generally accessible.

Obviously, this is a recipe for disaster. Less obviously, the problem is hardly limited to memcached. Most NoSQL stores have no concept of security. They’ll let anyone connect and fetch or overwrite any object. One of the best known doesn’t even check that input is well formed, so “cat /dev/urandom | nc $host $port” from anywhere would crash it quickly. Among all of the other differences between SQL and NoSQL systems – ACID, joins, normalization and referential integrity, scalability and partition tolerance, etc. – the near-total abandonment of security in NoSQL is rarely mentioned. Lest it seem that I’m throwing stones from some other garden, I’d have to say many filesystems hardly fare any better. For example, I generally like GlusterFS but it provides only the most basic kind of protection against information leakage or tampering. As a POSIX filesystem it at least has a notion of authorization between users, but it does practically nothing to authenticate those users and authorization without authentication is meaningless. The system-level authorization to connect is trivially crackable, and once I’ve done that I can easily spoof any user ID I want – including root. I’ve had to make the point over and over again in presentations that cloud storage in general – regardless of type – is usually only suitable for deployment within a single user’s instances, protected by those instances’ firewalls and sharing a common UID space. For most such stores, if a cloud provider wants to offer it as a public, shared, permanent service separate from compute instances, a lot more work needs to be done.

What kind of work? Mostly it falls into two categories: encryption and authentication/authorization (collectively “auth”). For encryption, there’s a further distinction to be made between on-the-wire and at-rest encryption. A lot of cloud-storage vendors make all sorts of noise about their on-the-wire encryption, but they stay quiet or vague about at-rest encryption and that’s actually more important. The biggest threat to your data is insiders, not outsiders. The insiders aren’t even going on the wire, so all of that AES-256 encryption there doesn’t matter a bit. Insiders should also be assumed to have access to any keys you’ve given the provider, so the only way you can really be sure nobody’s messing with your data is if you never give them unencrypted data or keys for that data. Your data must remain encrypted from the moment it leaves your control until the moment it returns again, using keys that only you possess. I know how much of a pain that is, believe me. I’ve had to work through the details of how to reconcile this level of security with multi-level caching and byte addressability in CloudFS, but it’s the only way to be secure. Vendors’s descriptions of what they’re doing in this area tend to be vague, as I said, but Nasuni is the only one who visibly seems to be on the right track. It sure would be nice if people could get that functionality through open source, instead of paying both a software and a storage provider to get it. Cue appearance by Zooko to plug Tahoe-LAFS in 5, 4, 3, …

The other area where work needs to be done is handling user identities, which covers both auth and identity mapping. For starters, the storage system must internally enforce permissions between users, which of course means it must have a notion of there even being multiple users. For systems which can assume that a single connection belongs to a single user, you can then authenticate using SASL or similar and be well on your way to a full solution. For systems that can’t make such an assumption, which includes things like filesystems, that’s not sufficient. You need to identify and authenticate not just the system making a request, but the user as well. I’m not a security extremist, so I can accept the argument that if you can fully authenticate a system and communicate with them through a secure channel then you can trust them to identify users correctly. The alternative is something like GSSAPI, which requires less trust in the remote system but can be a pretty major pain to implement.

The last issue is identity mapping. Even if you can ensure that a remote system is providing the correct user IDs, those IDs are still only correct in their context. If you’re a cloud service provider, you really can’t assume that tenant A’s user X is the same as tenant B’s user X. Therefore, you need to map A:X and B:X to some global users P and Q. Because you might need to store these IDs and then return them later (e.g. on a stat() call if you’re a filesystem) you need to be able to do the reverse mapping back to A:X and B:X as well. Lastly, because cloud tenants can and will create new users willy-nilly, you can’t require pre-registration; you need to create new mappings on the fly, whenever you see a new ID. This ends up becoming pretty entangled with the authentication problem because authentication information needs to be looked up based on the global (not per-tenant) ID, so this can all be a big pain but – again – it’s the only way to be secure.

To sum up, the lesson of go-derper is not that memcached is uniquely bad. Lots of systems are equally bad, and making them less bad is going to be hard, but it needs to be done before the other promises made by those systems can be realized. For a great many people, systems that are so totally insecure are useless, no matter what other wonderful functionality they might provide.

• Dark Roasted Blend on July 16
• BoingBoing on August 1
• Damn Cool Pics on August 2
• Inhabitat on August 6

I predict that it will show up on at least one more website I follow. In maybe a week or so, I’ll see it in print media for the first time. Another week or two after that, I’ll see it in the Boston Globe. That’s the usual pattern, anyway.

[root@fserver-1 repo]# yum install python-tornado
Loaded plugins: presto
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package python-tornado.noarch 0:1.0-2.fc15 set to be updated

(hundreds of lines of dependency stuff)

Transaction Summary
=====================================================================================
Install      13 Package(s)
Upgrade     161 Package(s)

Total download size: 156 M
Is this ok [y/N]: n

Is this OK? Are you kidding? Of course it’s not OK, especially when I can see that the list includes things like gcc, vim, and yum itself. I know how systems get broken, and that’s it. By way of contrast, let’s see how it goes with easy_install.

[root@fserver-1 repo]# easy_install tornado
Searching for tornado
Reading http://pypi.python.org/simple/tornado/
Reading http://www.tornadoweb.org/
Best match: tornado 1.0
Downloading http://github.com/downloads/facebook/tornado/tornado-1.0.tar.gz
Processing tornado-1.0.tar.gz
Running tornado-1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-6Wcauv/tornado-1.0/egg-dist-tmp-NEPqMm
warning: no files found matching '*.png' under directory 'demos'
zip_safe flag not set; analyzing archive contents...
tornado.autoreload: module references __file__
Adding tornado 1.0 to easy-install.pth file

Installed /usr/lib/python2.6/site-packages/tornado-1.0-py2.6.egg
Processing dependencies for tornado
Finished processing dependencies for tornado

Yeah, I see the appeal. On one hand, hours spent either rebuilding a broken system or debugging the problems that are inevitable when 161 packages get updated. On the other hand, Getting Stuff Done in about a minute. Yes, I tested, and the result does work fine with the packages/versions I already had. Still, though, having to do things this way is awful. It’s bad enough that there are still separate package managers for different Linux distros, but now programmers need to have several different package managers on one system just to install the libraries and utilities they need. Worse still, most of these language-specific package managers suck. None of them handle licensing, and few of them handle dependency resolution in any kind of sane way. One of the most popular Java package managers doesn’t even ask before downloading half the internet with no version or authenticity checking to speak of. Good-bye, repeatable builds. Hello, Trojan horses. I can see (above) the problems of having One Package Manager To Rule Them All, or of having dependency resolution be too strict, but there has to be a better way.

What if the system package manager could delegate to a language-specific package manager when appropriate (e.g. yum delegating to easy_install in my example)? Then the system package manager could save itself a lot of work in such cases, and also avoid violating the Principle of Least Surprise when installing in the “standard way” for the system yields different results than installing in the “standard way” for the language. There’d still be difficult cases when dependencies cross language barriers, but those are cases that the system package manager already has to deal with. I know there are a lot of details to work out (especially wrt a common format for communicating what’s wanted and what actually happened), possibly there’s even some fatal flaw in this approach, but my first guess is that a federation/delegation model is likely to be better than an everyone-conflicting model.

Before I start talking about the Four Horsemen of Poor Performance, it’s worth establishing a bit of context. Processors have actually not gotten a lot faster in terms of raw clock speed since 2002 – Intel was introducing a 2.8GHz Pentium 4 then – but they’ve all gone multi-core with bigger caches and faster buses and such. Memory and disk sizes have gotten much bigger; speeds have increased less, but still significantly. Gigabit Ethernet was at the same stage back then that 10GbE is at today. Java has gone from being the cool new kid on the block to being the grumpy old man the new cool kids make fun of, with nary a moment spent in between. Virtualization and cloud have become commonplace. Technologies like map/reduce and NoSQL have offered new solutions to data problems, and created new needs as well. All of the tradeoffs have changed, and of course we’ve learned a bit as well. Has any of that changed how the Four Horsemen ride?

With regard to data copies, I think we’ve lost ground. More people now realize that data copies are bad, but with processors and memory being so much faster they seem less inclined to do anything about it. Many “modern” languages have absolutely atrocious support for the kind of efficient buffer-list methods I recommended. Immutable-data languages inevitably force programmers to copy data into new variables where once they would have updated in place. Sometimes they even force the new variable to be local in a new function called only for that purpose. You can argue all you like about the concurrency or robustness advantages of the immutable-data approach, you can argue that good programmers won’t “subvert” your favorite language that way, but the fact is that real-world programmers do engage in such subversion and it does carry a performance cost. Even if much of that cost is ameliorated by using reference counts instead of true copies, it’s still less efficient than modifiable buffer chains.

The context-switch issue is the one being debated most nowadays, but surprisingly little has actually changed. Pretty much all of what I said earlier – about the ratio of threads to processors, about single-threaded approaches being beneath contempt, about coroutines giving all of the headaches of concurrency with none of the advantages – remains true. I guess that’s not all that much of a surprise since I’d been working on multiprocessors for years when I wrote the article even though they weren’t fully mainstream yet. Sure, the scalability of native threads has improved a lot, most notably in Linux, but context switches still aren’t free. In a multiprocessor system, you also have the problem of resuming a thread on a different processor with a stone-cold cache. Some people who just heard of cohort scheduling think it provides a silver-bullet solution, but it really doesn’t. If you want to worry about cache warmth, you have to think about three kinds of cached data.

• Instructions (yes, even with a unified I+D cache)
• The actual data being operated on.
• The secondary data representing request state, global/persistent data structures, etc.

Cohort scheduling mostly addresses the first of these, and somewhat the third. Inevitably, by preserving locality in these cases it tends to make things a bit worse for the second – and largest – category of cache contents. The paper was written in the same era as my own post. It made a certain amount of sense in the context of the type/size/speed of caches in use at the time, and the loads being placed on them. Does it make an equal amount of sense in today’s context? Maybe. Sometimes. There’s no silver-bullet solution here. If you really want to optimize for cache warmth, you’ll still have to think hard about what data is being cached, how it’s moving between cache levels, and what program structures will create access patterns that minimize that movement.

My conclusion, just as before: by all means use multiple native threads, and use multiple coroutines on top of those if it suits you, but use both judiciously and pragmatically. Above all, try to program in a way that maximizes your flexibility to adjust the balance between event-based and multi-threaded and coroutine-based approaches in your program, as the tradeoffs and the program itself continue to change.

On the last two issues – memory allocation and lock contention – there has also been little change. Memory allocation is becoming a bit of a hot issue as more people realize that their oh-so-convenient languages and frameworks are in some cases creating/deleting thousands of objects per request. That’s just obscene. The best response is to avoid the overhead altogether, going as far up the stack as you have to, but for those unwilling to do so there’s still ample opportunity to make those wasteful operations less costly. Similarly, there have been some advances in avoiding lock contention – most notably wider knowledge and acceptance of the actor model – but it remains a very thorny problem that I’d need a whole separate post to address.

Perhaps the biggest change I’d make to the original post, if I were writing it today, is to the grab-bag of items at the end. For example, many people are coming up with code to exploit the vast differences in latency and throughput between sequential and random disk I/O. Log-structured filesystems are relevant again, even if many insist on embedding ad-hoc, informally specified, bug-ridden, slow implementations (with apologies to Greenspun) in other systems before they’ve even read up on the original. There’s a fifth horseman there, which is the failure to account for the motion of data between levels of the storage hierarchy, or deal properly with the fundamental differences between those levels. Think about the following differences.

• Cache: extremely fast, seemingly byte-addressable but really more alignment-sensitive than people think, very limited size.
• Memory: slower than cache but faster than storage by an even greater degree, byte addressable
• Direct-connected SSD (e.g. Fusion): very alignment sensitive, low/flat latency but some artifacts due to wear leveling and garbage collection
• External SSD: a tiny bit slower than direct-connected, but otherwise similar
• Single disk: slower, more variable latency due to seeks (few people can deal with seek issues well enough to notice rotational latency any more), internal caches which affect both durability and performance ratios
• Disk array: faster than single disk but different ratios for read vs. write and sequential vs. random, cache is usually battery-backed and large enough that writes very rarely have to wait (but reads often do)

That’s a lot of differences, which can be hard to account for. Failing to understand these different characteristics and the consequences of translating between them is what’s behind misunderstandings about caches and cohort scheduling, behind those loathsome memory-only “data stores” and even behind a lot of SSD mania. In many cases this data motion is the primary determinant of overall program performance. It needs to be carefully managed to ensure that data will be where you want it, when you want it, with a minimum of time spent playing “musical chairs” with it.

First, I’ll say that hundred-slide decks annoy me. Yes, I know it’s usually a reaction to the problem of slides that are too few and too densely packed, leading to the also-awful phenomenon of the presenter spending most of the time just repeating what everyone can already read, but it’s an over-reaction. The other day I was reading some slides online, and I encountered the following pattern:

Slide N-1: (clip art)
Slide N: “vs.”
Slide N+1: (more clip art)

A whole slide just for “vs.”? That’s wasting my time. Presenters who use that style end up spending too much of their presentation actually changing slides and waiting the obligatory five seconds for the audience to catch up, no matter how little content is on each. Stephen Foskett pointed out that Lawrence Lessig only puts one word on each slide and is still a very highly regarded speaker. Well, yeah, he’s Lawrence Lessig. I’m not, you’re not, and probably neither is anyone you know (unless of course you know Lessig).

Now, I know presentation length can be tricky. I myself do tend to err on the side of making my slides too busy and very spare graphically. I do that because I know that the slides are likely to be viewed more in email etc. than with me actually presenting them, so to make sure they’re useful as a reference I often sacrifice a little on the “live” side. What I’d generally like to do is create two decks – one verbally spare and graphically rich to illustrate or anchor what I’m saying live, and a longer form for sending around later. That means even more time spent in Impress, though, and is often not feasible for various other reasons as well. My best advice is to determine a good “minutes per slide” figure based on the content, the audience, and an honest appraisal of your own ability to keep the audience interested while the slides aren’t changing, then use that to determine an appropriate slide count. If you’re a very dynamic speaker, you can go the Lessig route and spend five minutes on a one-word slide. If you need a hundred slides to fill a thirty-minute presentation, then maybe you’re admitting something about your speaking skills or the intrinsic value of what you’re presenting.

Second lesson: don’t get too cute. I’ve seen too many presentations lately, especially in the “edgier” tech areas, where the author had obviously spent way more time on finding funny clip art and quotes than on the actual content. Again, it’s a balancing act. Humor is good. A good quote or graphic can be an absolutely fantastic anchor for an important point, which you then elaborate or build on verbally. One not-really-funny slide after another after another with too little in between is just distracting.

Another error that I find even less excusable is simple ugliness. Yesterday I saw a presentation which had been done entirely – from title to closing – in what looked like a version of Comic Sans done to look like paint-brush strokes (house painting, not portrait painting). It wasn’t very readable, and looked totally amateurish. I was embarrassed for the author.

Now, somebody’s probably going to think I’m saying that I’ll totally dismiss an otherwise good presentation of an important idea because of slide count or graphics or font choice. Not so. I’ll still listen, but it will cost the author a “point” in my mind. It’s worth keeping in mind that, in these situations, every single point can matter. If you’re presenting to hundreds of people and only care if one or two respond in any significant way to what you’re saying, then maybe none of this matters. Far more presentations are given in smaller groups, though, where the opinion of everyone at the table does matter. People being how they are, they will use all sorts of nuances to form an impression of whether you’re smart, whether you’re trustworthy, etc. It probably won’t be one big thing that causes you not to get that next meeting, but an accumulation of little things. (If you think “meritocratic” open-source techies are any different, BTW, you’re just kidding yourself. The standards are different, but they’re just as stringently applied. Set the wrong tone and you’ll be written off just as surely and completely.) Why give someone the chance to think that you’re too serious or too frivolous, that your presentation shows disorganization, poor prioritization or disrespect for others’ time or sensibilities? Focus on content, by all means, but take just a little time to make sure it’s being delivered in a way that will ensure a good reception.