Bad Presentations

Presentations are the bane of the modern engineer’s existence. If you’re watching a presentation then it means you’re in a meeting, which is already something most of us don’t enjoy, and even worse it means you’re in a kind of meeting (or part of a meeting) that’s only minimally interactive. If you’re giving a presentation, that means even more time away from the technical tasks that drew you to this profession. Nonetheless, any project leader/advocate nowadays and for the last several years has had to spend a lot of time and energy on what is essentially a marketing activity, which is why I dubbed it “markelopment” (a deliberate riff on “devops”) on Twitter. I’m not among those who think presentations are always evil and should be shunned, but after having created/delivered quite a few presentations and sat through a great many more, I think I’m in a position to offer just a little bit advice.

First, I’ll say that hundred-slide decks annoy me. Yes, I know it’s usually a reaction to the problem of slides that are too few and too densely packed, leading to the also-awful phenomenon of the presenter spending most of the time just repeating what everyone can already read, but it’s an over-reaction. The other day I was reading some slides online, and I encountered the following pattern:

Slide N-1: (clip art)
Slide N: “vs.”
Slide N+1: (more clip art)

A whole slide just for “vs.”? That’s wasting my time. Presenters who use that style end up spending too much of their presentation actually changing slides and waiting the obligatory five seconds for the audience to catch up, no matter how little content is on each. Stephen Foskett pointed out that Lawrence Lessig only puts one word on each slide and is still a very highly regarded speaker. Well, yeah, he’s Lawrence Lessig. I’m not, you’re not, and probably neither is anyone you know (unless of course you know Lessig).

Now, I know presentation length can be tricky. I myself do tend to err on the side of making my slides too busy and very spare graphically. I do that because I know that the slides are likely to be viewed more in email etc. than with me actually presenting them, so to make sure they’re useful as a reference I often sacrifice a little on the “live” side. What I’d generally like to do is create two decks – one verbally spare and graphically rich to illustrate or anchor what I’m saying live, and a longer form for sending around later. That means even more time spent in Impress, though, and is often not feasible for various other reasons as well. My best advice is to determine a good “minutes per slide” figure based on the content, the audience, and an honest appraisal of your own ability to keep the audience interested while the slides aren’t changing, then use that to determine an appropriate slide count. If you’re a very dynamic speaker, you can go the Lessig route and spend five minutes on a one-word slide. If you need a hundred slides to fill a thirty-minute presentation, then maybe you’re admitting something about your speaking skills or the intrinsic value of what you’re presenting.

Second lesson: don’t get too cute. I’ve seen too many presentations lately, especially in the “edgier” tech areas, where the author had obviously spent way more time on finding funny clip art and quotes than on the actual content. Again, it’s a balancing act. Humor is good. A good quote or graphic can be an absolutely fantastic anchor for an important point, which you then elaborate or build on verbally. One not-really-funny slide after another after another with too little in between is just distracting.

Another error that I find even less excusable is simple ugliness. Yesterday I saw a presentation which had been done entirely – from title to closing – in what looked like a version of Comic Sans done to look like paint-brush strokes (house painting, not portrait painting). It wasn’t very readable, and looked totally amateurish. I was embarrassed for the author.

Now, somebody’s probably going to think I’m saying that I’ll totally dismiss an otherwise good presentation of an important idea because of slide count or graphics or font choice. Not so. I’ll still listen, but it will cost the author a “point” in my mind. It’s worth keeping in mind that, in these situations, every single point can matter. If you’re presenting to hundreds of people and only care if one or two respond in any significant way to what you’re saying, then maybe none of this matters. Far more presentations are given in smaller groups, though, where the opinion of everyone at the table does matter. People being how they are, they will use all sorts of nuances to form an impression of whether you’re smart, whether you’re trustworthy, etc. It probably won’t be one big thing that causes you not to get that next meeting, but an accumulation of little things. (If you think “meritocratic” open-source techies are any different, BTW, you’re just kidding yourself. The standards are different, but they’re just as stringently applied. Set the wrong tone and you’ll be written off just as surely and completely.) Why give someone the chance to think that you’re too serious or too frivolous, that your presentation shows disorganization, poor prioritization or disrespect for others’ time or sensibilities? Focus on content, by all means, but take just a little time to make sure it’s being delivered in a way that will ensure a good reception.

Twitter Quality Ratio

I just thought of a new metric: followers per tweet. I’m at about 0.43, which is pretty middle of the road. I see some people who are flirting with the 0.1 mark. At the other end of the scale I see some some who are at 3.0 or better. Not too surprisingly, the first group are disproportionately likely to end up on my “whale-jumpers” list which I check less often, and a similarly disproportionate number of my favorite tweeple seem to be in the second group. I could therefore expect to improve the “quality” of my own personal Twitter stream by checking this ratio for people I’m thinking of following . . . and you could do the same for your own personal stream as well.

BTW, if you want to help me pump up my own ratio, I’m @Obdurodon. ;)

The High Performance Cloud

Amazon has announced Cluster Compute Instances for EC2. This is a very welcome development. Having come from SiCortex, where we provided a somewhat cloud-like ability for users to allocate large numbers of very well-connected nodes on demand, I’ve been talking to people about the idea of provisioning cloud resources on special machines like this for at least the past year. In that light, I find a couple of things about the announcement a bit surprising. Let’s go down the specs first.

  • There’s a single type of HPC instance – dual quad-core “Nehalem” processors with 23GB. The Amazon page points out that this small additional amount of transparency about the exact CPU type allows people to do processor-specific optimization that they generally can’t do elsewhere in EC2.
  • Each instance comes with 1.7TB of instance storage. Performance is not mentioned, but at modern disk drive sizes that might well be just two drives.
  • Connectivity is via 10GbE (NIC and switch vendors not specified). Yuk. 10GbE still lags behind InfiniBand in terms of both bandwidth and latency, both absolute and per dollar. Much has been made lately of the significant and increasing dominance of IB in the HPC world, especially in the Top 500, and the customers Amazon is trying to attract are likely to consider 10GbE a strange choice at best.
  • There is a default limit of eight Cluster Compute Instances without filling out a request form. Eight machines is not enough for serious work of this type, even when the machines are this powerful, so that’s going to affect – and annoy – practically every user.
  • The instances are $1.60 per hour, which is $38.40 per day or a thousand a month. There are others far better qualified to comment on the economics, so I’ll leave it at that.
  • Cluster Compute Instances are only available in one availability zone.

My first thought is that the new offering as currently specified is nowhere near as interesting as it could be – and might be, as the service continues to evolve. Faster interconnects are one obvious way to make it more interesting. Removing the eight-machine default limit – which I strongly suspect is related to the capacity of the switches they’re using – is another. Then it gets even more interesting. When I’ve talked to people about heterogeneous clouds, which is what we’re heading towards here, I’ve generally meant far more kinds of specialization than this. How about instances which are optimized for communications instead of computation, such as with the same 10GbE (or better) but less powerful processors? How about instances which are optimized for disk I/O with multiple spindles and/or SSDs? How about special GPU-equipped instances? Once you can deal with the kind of heterogeneity that today’s CCIs represent, it’s but a short step to handling these other variations as well, so today’s announcement might merely foreshadow even bigger things to come.

The other thought I have about this is that it’s not just about the individual instances. The ability to specify that several instances should be provisioned close to another – probably on the same switch for reasons I mentioned above – is interesting with respect to both the user experience and the infrastructure needed to support it. Location transparency might be a defining feature of cloud computing, but that’s only in the sense of absolute location. Relative location is still a very valid parameter for allocation of cloud-computing resources. When you define a “cluster placement group” in EC2 you’re effectively saying that these instances should all be close to one another, regardless of where they all are relative to anything else. In other situations, such as disaster recovery, you might want to say certain instances should definitely be far from each other instead. We’ve been thinking through a lot of these issues on Deltacloud, but this isn’t a work blog so it would be both unwise and distasteful to say much more about that right now. Suffice it to say that facilitating this kind of placement requires a much more sophisticated cloud infrastructure than “grab whatever’s free wherever it is” which is pretty much the current standard. When you consider relationships not only between instances but also between instances and the data or connectivity they need, it can become quite a science project. The possibility that Amazon might be doing some of that science is, to me, one of the most exciting things about this announcement.

Stones in the Pond

I’ve been on vacation for the last few days, and while I was (mostly) gone a few interesting things seem to have happened here on the blog. The first is that, after a totally unremarkable first week, my article It’s Faster Because It’s C suddenly had a huge surge in popularity. In a single day it has become my most popular post ever, more than 2x its nearest competitor, and it seems to have spawned a couple of interesting threads on Hacker News and Reddit as well. I’m rather amused that the “see, you can use Java for high-performance code” and the “see, you can’t…” camps seem about evenly matched. Some people seem to have missed the point in even more epic fashion, such as by posting totally useless results from trivial “tests” where process startup dominates the result and the C version predictably fares better, but overall the conversations have been interesting and enlightening. One particularly significant point several have made is that a program doesn’t have to be CPU-bound to benefit from being written in C, and that many memory-bound programs have that characteristic as well. I don’t think it changes my main point, because memory-bound programs were only one category where I claimed a switch to C wouldn’t be likely to help. Also, programs that store or cache enough data to be memory-bound will continue to store and cache lots of data in any language. They might hit the memory wall a bit later, but not enough to change the fundamental dynamics of balancing implementation vs. design or human cycles vs. machine cycles. Still, it’s a good point and if I were to write a second version of the article I’d probably change things a bit to reflect this observation.

(Side point about premature optimization: even though this article has been getting more traffic than most bloggers will ever see, my plain-vanilla WordPress installation on budget-oriented GlowHost seems to have handled it just fine. Clearly, any time spent hyper-optimizing the site would have been wasted.)

As gratifying as that traffic burst was, though, I was even more pleased to see that Dan Weinreb also posted his article about the CAP Theorem. This one was much less of a surprise, not only because he cites my own article on the same topic but also because we’d had a pretty lengthy email exchange about it. In fact, one part of that conversation – the observation that the C in ACID and the C in CAP are not the same – had already been repeated a few times and taken on a bit of a life of its own. I highly recommend that people go read Dan’s post, and encourage him to write more. The implications of CAP for system designers are subtle, impossible to grasp from reading only second-hand explanations – most emphatically including mine! – and every contribution to our collective understanding of it is valuable.

That brings us to what ties these two articles together – besides the obvious opportunity for me to brag about all the traffic and linkage I’m getting. (Hey, I admit that I’m proud of that.) The underlying theme is dialog. Had I kept my thoughts on these subjects to myself or discussed them only with my immediate circle of friends/colleagues, or had Dan done so, or had any of the re-posters and commenters anywhere, we all would have missed an opportunity to learn together. It’s the open-source approach to learning – noisy and messy and sometimes seriously counter-productive, to be sure, but ultimately leading to something better than the “old way” of limited communication in smaller circles. Everyone get out there and write about what interests you. You never know what the result might be, and that’s the best part.

(Dedication: to my mother, who did much to teach me about writing and even more about the importance of seeing oneself as a writer.)

MongoDB Durability

After writing recently about some of the less-savory tactics that are often used in positioning technical products/projects, another ugly example of FUD has raised its head in the NoSQL world. No, I’m not referring to the “Facebook is abandoning Cassandra” silliness, which I think Eric Evans dealt with quite adequately. I’m talking about the MongoDB will throw away your data silliness, where somebody involved in a competing project expresses “sincere concern” about data durability. Mikeal actually raises some excellent points about data durability. I have worked for many years in environments where such things are taken very seriously, and he’s correct on many of the technical points such as the inadequacy of mmap-based approaches for ensuring recoverability. I do like MongoDB’s feature set, which has led me to use it for one project, but I admit that my faith in their ability and intention to fix some of the durability issues is just that – faith. It’s faith based on knowing something about the people involved, and knowing – contra Mikeal – that the problems do not run so deep as to be unsolvable without a major overhaul, but it’s faith rather than current reality. Mikeal’s points would be better made without some of the exaggerations and misrepresentations that others have taken pains to correct in the comments, and especially without remarks like this:

Using an append-only file is the preferred, sane and most assured way to handle data loss or corruption.
[after being corrected and reminded of the soft-update alternative]
while soft updates are safe their inventor and implementor in UFS have admitted are very hard to get right.

“Hard to get right” and “insane” are not the same. Some things are intrinsically hard to get right but are still worth doing. That includes all of the hard work kernel people do on placement/alignment and ordering/flushing to make sure those append-only files work so well even for systems-programming tyros. It also includes some of the compaction/vacuuming problems that append-only files introduce (closely related to “segment cleaning” in log-structured filesystems). Two of the best-respected filesystems out there – ZFS and btrfs – are primarily based on COW rather than append-only logs/journals, though ZFS does have an intent log as well. VoldFS also uses COW, and I’ll gladly debate Mikeal on the merits of my “insane” choice for that environment or use case.

The claim that the approach used by CouchDB is the only sane and assured one doesn’t help anyone. It’s merely partisan, not constructive. As Kristina Chodorow points out in the comments, it’s irritating when somebody just says “MongoDB ate my data” or “CouchDB is slow” without providing any specifics that can be addressed. (Since Mikeal seems to feel differently, I’m sure he won’t mind me mentioning that “CouchDB is a hog” is exactly the response I got when I suggested using it to replace MongoDB in that afore-mentioned project because I prefer its replication model.) As Dwight Merriman is also quoted in the comments as saying, the days of one-size-fits-all storage are over. MongoDB’s choices and roadmap might not suit everybody. Neither do CouchDB’s. There’s nothing wrong with proponents of one project engaging in constructive dialog regarding issues in others, but it does help if such criticism is in fact constructive and if people don’t consistently offer far more or sharper criticism than they are willing to accept themselves.

It’s Faster Because It’s C

I was recently drawn into another discussion about a claim that project Foo was faster than project Bar because Foo is written in C (or maybe C++) and Bar is written in Java. In my experience, as a long-time kernel programmer and as someone who often codes in C even when there are almost certainly better choices, such claims are practically always false. The speed at which a particular piece of code executes only has a significant effect if your program can find something else to do after that piece is done – in other words, if your program is CPU-bound and/or well parallelized. Most programs are neither. The great majority of programs fit into one or more of the following categories.

  • I/O-bound. Completing a unit of work earlier just means waiting longer for the next block/message.
  • Memory-bound. Completing a unit of work earlier just means more time spent thrashing the virtual-memory system.
  • Synchronization-bound (i.e. non-parallel). Completing a unit of work earlier just means waiting longer for another thread to release a lock or signal an event – and for the subsequent context switch.
  • Algorithm-bound. There’s plenty of other work to do, and the program can get to it immediately, but it’s wasted work because a better algorithm would have avoided it altogether. We did all learn in school why better algorithms matter more than micro-optimization, didn’t we?

If you look at this excellent list of performance problems based on real-world observation, you’ll see that most of the problems mentioned (except #5) fit this characterization and wouldn’t be solved by using a different language. It’s possible to run many synchronization-bound programs on one piece of hardware, with or without virtualization, but the fewer resources these programs share the more likely it becomes that you’ll just become memory-bound instead. On the flip side, if a program is purely disk-bound or memory-bound then you can obtain more of those resources by distributing work across many machines, but if you don’t know how to implement distributed systems well you’ll probably just become network-bound or synchronization-bound. In fact, the class of programs that exhibit high sensitivity to network latency – a combination of I/O-boundedness and synchronization-boundedness – is large and growing.

So, you have a program that uses efficient algorithms with a well-parallelized implementation, and it’s neither I/O-bound nor memory-bound. Will it be faster in C? Yes, it very well might. It might also be faster in Fortran, which is why many continue to use it for scientific computation but that hardly makes it a good choice for more general use. Everyone thinks they’re writing the most performance-critical code in the world, but in reality maybe one in twenty programmers are writing code where anything short of the most egregious bloat and carelessness will affect the performance of the system overall. (Unfortunately, egregious bloat and carelessness are quite common.) There are good reasons for many of those one in twenty to be writing their code in C, but even then most of the reasons might not be straight-line performance. JIT code can be quite competitive with statically compiled code, and even better in many cases, once it has warmed up, but performance-critical code often has to be not only fast but predictable. GC pauses, JIT delays, and unpredictable context-switch behavior all make such languages unsuitable for truly performance-critical tasks, and many of those effects remain in the runtime libraries or frameworks/idioms even when the code is compiled. Similarly, performance-critical code often needs to interact closely with other code that’s already written in C, and avoiding “impedance mismatches” is important. Most importantly, almost all programmers need to be concerned with making their code run well on multiple processors. I’d even argue that the main reason kernel code tends to be efficient is not because it’s written in C but because it’s written with parallelism and reentrancy in mind, by people who understand those issues. A lot of code is faster not because it’s written in C but for the same reasons that it’s written in C. It’s common cause, not cause and effect. The most common cause of all is that C code tends to be written by people who have actually lived outside the Java reality-distortion bubble and been forced to learn how to write efficient code (which they could then do in Java but no longer care to).

For those other nineteen out of twenty programmers who are not implementing kernels or embedded systems or those few pieces of user-level infrastructure such as web servers (web applications don’t count) where these concerns matter, the focus should be on programmer productivity, not machine cycles. “Horizontal scalability” might seem like a euphemism for “throw more hardware at it” and I’ve been conditioned to abhor that as much as anyone, but hyper-optimization is only a reasonable alternative when you have a long time to do it. Especially at startups, VC-funded or otherwise, you probably won’t. Focus on stability and features first, scalability and manageability second, per-unit performance last of all, because if you don’t take care of the first two nobody will care about the third. If you’re bogged down chasing memory leaks or implementing data/control structures that already exist in other languages instead of on better algorithms or new features, you’re spending your time on the wrong things. Writing code in C(++) won’t magically make it faster where it counts, across a whole multi-processor (and possibly multi-node) system, and even if it did that might be missing the point. Compare results, not approaches.

Hey Reddit users, if you want to try something less than two years old, how about today’s post? Thanks!

Converts, Turncoats, and Moles

Everyone knows that the market for products can be very competitive, and that the competition can often take on a distinctly shady character. What seems to be less appreciated is that the “marketplace of ideas” can be just as competitive, and often just as shady. The battles for “mind share” between one project and another, or even between one approach and another, can be fierce. At the more obviously commercial end of this spectrum are debates like NAS vs. SAN, 10GbE vs. IB, iSCSI vs. FCOE, end-to-end vs. embedded functionality. Often the commercial interests and biases of the participants are obvious, but other times less so. When Joe Random Blogger weighs in on one of these debates, finding out that Joe spent months and thousands of dollars to become a CCIE might shed some light on their vested interest. Other times, there’s no such obvious marker but the vested interest is just as real. This is why Stephen Foskett and others have called for explicit disclosure of such interests by bloggers. Full disclosure: I’ve met Stephen, I like Stephen, we’ve done an Infosmack podcast together and he invited me to drop by at a Tech Field Day where I got to enjoy some free food at EMC’s expense. See how easy that was?

Where things get a little murkier is where there’s not an obvious product involved, or where some of the projects are non-commercial. Some of the participants in the fierce SQL vs. NoSQL debates have a commercial stake, some have a personal stake, and some really have no stake at all beyond a desire to see open discussion advance the state of the art. Quick: which category do I fit into? I’m not even sure myself. I like to think I fit into the last category, but this stuff is not entirely unrelated to the job that puts food on my family’s table so arguments that I belong in one of the other categories have plenty of merit. It’s funny that in my day job I rarely get exposed to such conflicts of interest but as a blogger I do. I’ve been asked to write “objectively” about certain products or projects, or sometimes to refrain from writing about them. I’ve tried to ignore those requests as much as possible, and just write about what interests me . . . but I digress. What are we to make of sniping e.g. between Cassandra and HBase advocates, for example, when both are open source? At an even more abstract level, what about centralized metadata vs. “floating master” vs. peer-to-peer distribution, or using the same vs. separate algorithms for wide-area and local replication? What about public cloud vs. private cloud and the people who claim one or another is beneath contempt? The resolutions of such debates have clear implications for certain projects, and the participants are hardly unaware of that, but the debates are not directly about the projects.

All of this creates an environment ripe for manipulation by the less ethical. Here on this blog I’ve often posted about apparent instances of FUD and astroturf, which are two forms of such manipulation. Bloggers or Twitterers with undisclosed interests in one side of the debate are everywhere. I’ve seen one fellow with a vested interest in certain NoSQL projects repeatedly bash other projects quite savagely, more than once, for faults that his own pet projects still have or had until only a week before, without disclosing his direct involvement in the alternatives that remain after the bashing is done. One of the dark sides of open source is the practice of mining competitors’ code for flaws not so they can be fixed but so that they can be used as ammunition in the war of ideas. Perhaps the most effective technique I’m aware of in this area, though, is the wooing of converts. As much as we all like to pride ourselves on being completely rational and empirical, technology is still a social enterprise and nothing can help one side of a debate more than winning over a prominent member of the other team. “I used to believe in SAN/embedded/centralized but I’ve seen the error of my ways” can be very powerful. It strongly implies that the evolution from novice to expert and that from one position to the other are somehow linked. Novices are fooled into believing X, but experts have figured out Y. Sometimes I’m sure the change in heart is legitimate and sincere, driven by increased knowledge just as it seems or perhaps by the ever-changing tradeoffs we all have to make. (I’ve done this myself, with regard to storage vs. processing networks and the balance of traffic between the two in a distributed filesystem.) Other times, I’m just as sure that someone’s change of heart is the result of deliberate persuasion. Contact might have been deliberately made, perhaps through a mutual friend. The strongest features, situations, and (perhaps non-obvious) future plans/directions for one alternative might have been shared, all deliberately planned but presented as innocent exchange of ideas between colleagues. Sometimes a convert can be won this way, and since nobody is ever as zealous as a new convert the result can often be advocacy of the new preference even in situations where the old preference remains objectively better. Yes, you can buy publicity like that. What you can also do is create “moles” who gain some level of notoriety within a community – it’s really not that hard with emerging technologies – and then very noisily “defect” to an opposing camp.

I don’t know for sure how prevalent this sort of manipulation is. I’ve certainly seen plenty of astroturf and FUD, I’ve seen some of the attempts to persuade “thought leaders” one way or another, but I don’t know for sure if I’ve ever actually seen a mole. What I do know is that I don’t see everything, and I’d be a fool to believe these things don’t happen. The means, motive, and opportunity are all there. There are “social media experts” who are paid – and paid quite well – to do almost exactly what I’ve described; I’m sure not all of them have 100% clean hands. I’ve yet to meet a VP of marketing at a startup who would have any qualms, who would hesitate one second, over paying someone to do these things if they thought that person had the capability. I’m not saying we should all start jumping at shadows, but if you see a prominent advocate of low-cost open-source scale-out solutions suddenly start singing the praises of a vendor who is notoriously opposed to all three features, maybe you should at least consider the possibility that their change of position is something other than a total accident.