Dumb Things People Say

So good old @dozba, in his typically endearing style, wrote about why MacOS X is an Unsuitable Platform for Web Development. His criticism about Textmate seems a bit silly, but his rants about package management remind me of my own problems with a proliferation of language-specific package systems, and “you don’t deploy to BSD” is spot-on. You would think that people would realize that developing on one platform for deployment on another can be problematic at best, but apparently the lesson needs repeating. At work, we even see glitches when people develop RHEL stuff on Fedora or vice versa, and those are both far more similar than the Mac is to anything you would ever deploy in production. What was even more amazing to me was the Slashdot reaction. Even knowing – all too well – how far Slashdot has declined, some of the stupidity on parade there literally made my jaw drop. Let’s review the main point here, OK?

Developing and deploying on different platforms is only OK if you stay well inside some kind of insulated sandbox and don’t care about performance.

The first part, about sandboxes, is mostly about avoiding functional differences between the platforms. If you ever use platform-specific configuration info or ioctls or anything else that requires an additional abstraction layer to make the code even run on both platforms, then you’re on very thin ice. Nine times out of ten you’ll be giving the people who actually deploy the code on real systems a good reason to hate you. As for performance, there are two sub-points. If you’re running in that nice insulated sandbox of a JVM, then its performance warts with regard to thread and memory handling will probably outweigh the warts in the OS scheduler and virtual memory systems. To a lesser extent the same might be true if you’re using Python or Ruby or Erlang. If the performance that matters is network or storage (filesystem/LVM/disk) performance, then you won’t even have that excuse. You just can not expect two operating systems from completely different families to exhibit the same balance and inflection points at different I/O rates, sizes, thread counts, etc. and all of the knobs for tuning around these idiosyncrasies will be different too. If performance matters at all for your application, you need to be working with the same performance ratios and tuning facilities across development and production. Does the Slashdot crowd seem to get any of this? Of course not.

Of course, you can dual-boot Linux on it or run it in VMWare. But you knew that, right?

Couldn’t be more wrong. If you’re doing anything virtual, then you’ve just made the matching-environment problem even worse, because now you’re subject to differences in both the host and guest OSes (and the hypervisor between them). Also, if you’re so addicted to running stuff in virtual machines then you’ll probably have a lot of them contending with one another and distorting the performance picture even more.

I can’t imagine writing code so finicky and unstable that it can only be cajoled into running under such a specific environment.

This guy has obviously never written anything but toy code. In the real world it’s very easy for code to run well on one platform, run poorly on others, and not run at all on still others. All you need to do is use one platform-specific feature, or rely on one platform-specific aspect of performance to guide your implementation tradeoffs. It’s writing code that runs on many platforms that can be a challenge, and code that runs well on many platforms is quite rare.

OK, so picking on Slashdotters is like shooting pithed fish in a very small bucket. The real point, though, is that all those developers using their MacBook Pros to develop code that’s supposed to run on Linux are doing their colleagues and users a great disservice. You don’t like the Linux environment as much? Too bad. Make it better. That’s how your deployment platform got to the point where you’re using it, after all. People used it, mostly liked it, and fixed/replaced the things they didn’t like so much, and added new stuff to scratch their own itches. If you use the same platform for development and deployment, then every improvement you make has twice as many chances to benefit someone else, and every improvement someone else makes has twice as many chances to benefit you.


Even before dot-bomb, I’d learned that “entrepreneur” meant two distinctly different things to two distinctly different groups of people. To one group, it meant an activity; to the others it meant an identity. It’s the difference between doing what entrepreneurs have historically done (mostly inventing things) and being like entrepreneurs have historically been – or, even more cynically, appearing like they have historically appeared. Having spent more time in startups than the vast majority of those who are or call themselves entrepreneurs, I feel a great deal of kinship with the first group. The second group? Not so much. These are the folks who come across as MBA types even if they actually have technical degrees (and sometimes non-trivial technical abilities). They dress like they believe entrepreneurs should dress, whether that fashion calls for suits or pure grunge or some blazer-and-jeans hybrid in any given month. They eat at the restaurants and drink at the bars that they think entrepreneurs are supposed to. They read the books and blogs that they believe entrepreneurs should read, they put in the hours that they believe entrepreneurs should put in (even if those hours are mostly on Twitter), they gravitate toward the office locations and layouts that they believe entrepreneurs should prefer. It’s like the old saying:

Sincerity is the key. If you can fake that, you’ve got it made.

Individuality is valued, so these folks try to cultivate it . . . but they all cultivate it the same way and end up looking more conformist than ever. Nowhere is that more apparent than in their adoption of entrepreneur jargon. Some of that jargon, such as that associated with VC/angel term sheets and such, actually has some grounding in reality. Other terms don’t, and the ones that have really been annoying me of late are “pivot” and “iterate”. I know all too well how important it is for a startup to be adaptable, but pivots and iterations are still ways of recovering from mistakes. They’re still something to avoid, not something to strive for. On average, the interval between such “about face” maneuvers should be longer than the interval between founding and whatever exit you prefer. It’s nice when a figure skater recovers from a botched landing, but if they spend their whole routine proving that ability then they’re not going to win a lot of titles. Somehow, though, many entreposeurs seem to think that being able to jump from one product idea to another is more important than being smart enough not to chase such ephemeral pseudo-markets, or executing quickly enough not to miss the boat on the first idea, or having the courage/honesty to admit that you goofed and move on to the next thing. Failure is not supposed to be a liability, right? We learn from failure, if you’re not willing to fail you’re not thinking big enough, yadda yadda . . . or maybe that was last year’s management-consultant money machine, because this year it’s apparently preferable to keep flailing around for something – anything! – that will keep the founders from having to get new business cards. After all, “founder of X” is their identity, couldn’t possibly let go of that now, could we? The irony is that I will almost certainly get comments from people who say the value is all in the team anyway, even though when the issue is inflated social-media-startup valuations those very same people will immediately forget the team and justify their paychecks solely by making up stuff about potential markets. It’s over here, no it’s over there, why cheat people out of dollars with three-card monte when you can play the same game for millions in Silly Valley?

Here’s a hint: don’t try to be an entrepreneur before you’ve generated any wealth for anybody. It doesn’t work. The way to be an entrepreneur is to have an idea you’re willing to stick with, and then build a business around it. If you have to put aside the starving-artist chic and actually work in the corporate trenches until you have some clue which ideas are actually worth pursuing (and haven’t already been done), so be it. That’s how every real entrepreneur I’ve known did it.

Morning in Tempe

I’m in Tempe for the Fedora Users and Developers Conference, a.k.a. FUDCon. Here are some random thoughts.

  • Enhanced pat-downs aren’t so bad.
  • The weather’s nice. I should have expected the palm trees, but I totally didn’t expect to see orange trees with ripe fruit hanging just out of arm’s reach (because the ASU students picked everything lower already).
  • The ASU campus is much more interesting and varied architecturally than any other campus I’ve been on. Sure, the color palette is a bit limited – light brown, dark brown, reddish brown – but the shapes and textures make up for it. Actually there was one nice splash of color, which was a gigantic wild rose bush clinging to the side of one building. That ugly bump just north of campus doesn’t do much for me, though.
  • I haven’t seen a single squirrel on campus. I did see two cats, though – fluffy persians who must be very uncomfortable in all this heat. I’ve seen and heard lots of unfamiliar birds, too – mostly grackles, I think
  • Meeting people in person is great. The Fedora crowd is notably casual, international, and friendly – even by technical-conference standards, in all three regards. I’d particularly like to thank Robyn Bergeron and Seth Vidal, very busy leaders in that community who have nonetheless gone out of their way to make me feel welcome and included. It was also especially nice to meet Pete Zaitcev and Major Hayden, because we’ve interacted so much online but never met until now.

Here’s a Flickr set for some of the pictures I’ve taken while here. OK, enough of the fluff. What about the real stuff? More bullet points, because that’s how I roll.

  • The whole “Bar Camp” style of pitching and voting on sessions was new to me. It did seem to work, though.
  • The first talk I attended was Marek Goldmann talking about BoxGrinder. I was pretty familiar with this work from my own involvement with Deltacloud/Aeolus, but Marek deserves kudos for presenting it well and even giving a live demo.
  • After lunch, it was Steven Dake talking about Sheepdog. Again, it’s work I’m familiar with. I think Steven and I will never quite agree on the value/importance of Sheepdog. On the one hand, the notion of distributed block storage has been very appealing to me for a long time. It’s why I went to Conley in 1998, and worked on C3D at EMC a few years later. On the other hand, block storage using a single specialized application interface which isn’t even as complex as the real system-level block device interface seems a bit unambitious to me. It just limits the applicability of the result too much IMO, and that seems a meager payoff for all that work solving the harder distributed-data problems. Of course, in this case it’s all NTT’s effort anyway. As far as the talk, a comparison to RBD would have been nice since anybody who’s interested in one should definitely check out the other as well.
  • Next up was Mike McGrath, talking about how cloud computing is going to displace non-cloud computing. Even as somebody who’s working on cloud stuff, I’m a little bit skeptical. Still, it was a good talk to get people thinking about all the implications.
  • I’ll skip the next talk, since it was mine and I’ll have more to say below.
  • The last talk of the day, for me, was Chris Lalancette talking about cloud management – especially Deltacloud and Aeolus. Having worked for a while on this project (and sitting about twenty feet from Chris most days) this was also pretty familiar territory, and Chris did a good job presenting on a complex subject. I apologize to both him and to Tobias Kunze (with whom I had an awesome chat later in the evening BTW) for putting them on the spot about the relationship between Makara and Aeolus.

So, how did my own presentation go? Somebody pointed out that I’d seemed a bit on edge the night before. Partly that was just the stress of travel and of being an introvert mingling with an unfamiliar group of people, but there’s another factor that I hadn’t even consciously realized until I was writing this post. I’ve presented about CloudFS privately and/or in fairly abstract terms so many times that I’d actually forgotten this was the first truly public presentation about a concrete thing that I’ll actually be delivering in the near future. That’s a big deal. I was a bit concerned at first because they’d put me in the largest room and at five past the hour it was still three-quarters empty. Nobody likes talking to an empty room. Shortly after I started, though, the room was pretty much full – not standing-room-only full, but I don’t remember seeing many empty seats. Not that I was trying too hard to count, of course; I was otherwise occupied. Even better, people were engaged. There were many questions, and they were good questions – questions that to me indicated genuine curiosity and constructive intent, not just the “I’m going to prove I’m smart” or “if you don’t get this one right your project will look silly” kinds of questions that one often gets. The post-presentation chatter even went on so long that Chris had to kick us away from the lectern. Good problem to have. :)

The best part of all, in my opinion, was outside of the talk itself. In at least two other presentations, and in even more hallway conversations, the possibility of using CloudFS to solve some problem or add some functionality came up. Also, at least one person had clearly given the code a pretty detailed look since my talk, asking questions and making comments about internal details that he could not have known about otherwise. That is so cool. It’s all very well to have people’s attention for an hour or so before people move on to the next new thing, but when something you’ve talked about shows up in colleagues’ own thinking about how to solve their own problems that’s an even surer measure of being on the right track. Thank you, everyone, for letting me be part of the broader progress we’re all making together.

CloudFS on the Horizon

I try to keep blogging about $dayjob to an absolute minimum, but this is kind of a big deal. Today, I pushed the first bits of CloudFS out to a public git repository. A week or so ago I registered cloudfs.org to hold non-code content, but it’s not really up yet so for now you can clone that repository and look at the README.md therein, or at the Fedora 15 feature page. Here are a few nuggets to get you started.

  • CloudFS is basically GlusterFS enhanced to allow deployment by a provider as a permanent shared service, rather than as a private thing that users can run within their own compute instances.
  • The enhancements necessary fall into several broad categories: authentication, encryption, isolation (each tenant gets their own namespace and UID/GID space), quota/billing, and some necessary enhancements to existing GlusterFS functionality.
  • This is a very pragmatic and unambitious release, explicitly not including the improved-DHT and multi-site-replication functionality that I think will make CloudFS really cool. Think of it as a warm-up to the main attraction.
  • The code is nowhere near complete. The three translators I’ve written are complete enough to do useful things – and more importantly to be worth reviewing – but all need to be improved in various ways and there are other bits (mostly around configuration and management) that don’t even exist yet. To put it another way, I think the code represents that point on a journey where you’ve climbed the highest hill and can see the destination, but there are many miles yet to be walked.

Once I get cloudfs.org set up the rest of the way, I’ll probably start posting more info there. Stay tuned.

Good Programmers

When you hear that someone’s a good programmer, what do you think that means?

  • They write lots of code.
  • They write good code (e.g. clear logic , good error checking, etc.)
  • They write difficult code, that’s difficult to get right or debug when it goes wrong.
  • They write innovative code, representing new algorithms or techniques.

All of the above, you say? Sorry, you don’t get all of the above. An average programmer can do two of these things, but not at the same time. That means nearly half can’t even do that well. A pretty good programmer can do three, two at a time. A very good programmer can do all four, perhaps even three at a time. There are programmers who can do all four at once, but they’re like people who can juggle eight balls at once – you’ll probably never meet one.

My point here is not to say programmers suck, or that one combination of abilities is better than another. The real point is that programmers can be good in different ways, even if none of them are perfect, and if you want to get the most from a group of programmers it can really help to recognize these differences. Programmers with one set of strengths love to dump all over programmers with a different set, but specialization is the cornerstone of our profession (if not of the entire economy that supports it) and specialization cuts across styles as well as technical areas. If you can get your specialists to complement and reinforce one another, instead of undermining and alienating one another, you can do great things that no band of generalists could match. Don’t celebrate diversity for its own stupid sake; this ain’t no hippie commune. Exploit diversity to get things done. It happens to be true that people are happier when they’re working to their strengths instead of conforming to some arbitrary “jack of all trades and master of none” standard, but that’s just icing on the cake. Being successful makes people happy too.

Friday Feature: Latest Tweets

As always, follow @Obdurodon for more.

  • It’s an antipattern to use “it’s an antipattern” as the worst insult you can think of for something that’s not a pattern.
  • OH: “I’d run Eclipse to take a look, but I want my machine to have some resources left for real work.”
  • I’m going to stop talking about cloud storage. Clouds are ephemeral. OceanStore had the right naming idea.
  • Every time a programmer uses goto, an architect dies a little. Die, you bastard, die!
  • If you want authority, take responsibility.

I Love You, But…

I love programming. I love thinking about algorithms and data strucures. I love writing code, rearranging code, talking about code. I even love testing and debugging and documenting code. (This is not to say I do all of these things as consistently as I can. There are still only 24 hours in a day and so one must prioritize.) Sometimes I think of getting out of this field, though, because so much of working as a programmer nowadays has nothing to do with any of the things I love, and it seems to be getting worse. Nobody loves meetings and bureaucracy and such, but that’s not what I’m talking about.

I hate spending half my time dealing with build systems, source-control systems, package managers, and such. There are too many out there, they all suck, everybody has their favorite one and their favorite way of using it, and they’re not at all shy about ramming their preferences down your throat . . . which brings me to my real point. I hate programmers. Hot damn, but we are a noxious breed, aren’t we? I’m tired of the backstabbing, the trashing each others’ work, the holier-than-thou attitude from the GNU types, the rampant sexism, the bike-shedding, the endless effort to do and re-do all the fun stuff while dumping as much work as possible onto one’s peers, and on and on and on. I know I’ve exemplified many of these sins myself, I don’t need anyone else to tell me that, but if I made it my life’s goal to be as much of a jerk as possible I’d still find myself outdone just about every day by people who aren’t even at their worst.

Of course, I don’t know what else I’d do that pays the bills, so you all are stuck with me, but come on, people. Let’s stop sucking all the fun out of this profession.

Language Specific Package Managers

As many people I’ve talked to IRL probably know, I really hate language-specific package managers. Java has several, Python/Ruby/Erlang etc. each have their own, etc. I totally understand the temptation. I know it’s not all about NIH Syndrome (though some is); some of it’s about Getting Stuff Done as well. Consider the following example. I tried to install Tornado using yum.

[root@fserver-1 repo]# yum install python-tornado
Loaded plugins: presto
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package python-tornado.noarch 0:1.0-2.fc15 set to be updated

(hundreds of lines of dependency stuff)

Transaction Summary
Install      13 Package(s)
Upgrade     161 Package(s)

Total download size: 156 M
Is this ok [y/N]: n

Is this OK? Are you kidding? Of course it’s not OK, especially when I can see that the list includes things like gcc, vim, and yum itself. I know how systems get broken, and that’s it. By way of contrast, let’s see how it goes with easy_install.

[root@fserver-1 repo]# easy_install tornado
Searching for tornado
Reading http://pypi.python.org/simple/tornado/
Reading http://www.tornadoweb.org/
Best match: tornado 1.0
Downloading http://github.com/downloads/facebook/tornado/tornado-1.0.tar.gz
Processing tornado-1.0.tar.gz
Running tornado-1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-6Wcauv/tornado-1.0/egg-dist-tmp-NEPqMm
warning: no files found matching '*.png' under directory 'demos'
zip_safe flag not set; analyzing archive contents...
tornado.autoreload: module references __file__
Adding tornado 1.0 to easy-install.pth file

Installed /usr/lib/python2.6/site-packages/tornado-1.0-py2.6.egg
Processing dependencies for tornado
Finished processing dependencies for tornado

Yeah, I see the appeal. On one hand, hours spent either rebuilding a broken system or debugging the problems that are inevitable when 161 packages get updated. On the other hand, Getting Stuff Done in about a minute. Yes, I tested, and the result does work fine with the packages/versions I already had. Still, though, having to do things this way is awful. It’s bad enough that there are still separate package managers for different Linux distros, but now programmers need to have several different package managers on one system just to install the libraries and utilities they need. Worse still, most of these language-specific package managers suck. None of them handle licensing, and few of them handle dependency resolution in any kind of sane way. One of the most popular Java package managers doesn’t even ask before downloading half the internet with no version or authenticity checking to speak of. Good-bye, repeatable builds. Hello, Trojan horses. I can see (above) the problems of having One Package Manager To Rule Them All, or of having dependency resolution be too strict, but there has to be a better way.

What if the system package manager could delegate to a language-specific package manager when appropriate (e.g. yum delegating to easy_install in my example)? Then the system package manager could save itself a lot of work in such cases, and also avoid violating the Principle of Least Surprise when installing in the “standard way” for the system yields different results than installing in the “standard way” for the language. There’d still be difficult cases when dependencies cross language barriers, but those are cases that the system package manager already has to deal with. I know there are a lot of details to work out (especially wrt a common format for communicating what’s wanted and what actually happened), possibly there’s even some fatal flaw in this approach, but my first guess is that a federation/delegation model is likely to be better than an everyone-conflicting model.

Bad Presentations

Presentations are the bane of the modern engineer’s existence. If you’re watching a presentation then it means you’re in a meeting, which is already something most of us don’t enjoy, and even worse it means you’re in a kind of meeting (or part of a meeting) that’s only minimally interactive. If you’re giving a presentation, that means even more time away from the technical tasks that drew you to this profession. Nonetheless, any project leader/advocate nowadays and for the last several years has had to spend a lot of time and energy on what is essentially a marketing activity, which is why I dubbed it “markelopment” (a deliberate riff on “devops”) on Twitter. I’m not among those who think presentations are always evil and should be shunned, but after having created/delivered quite a few presentations and sat through a great many more, I think I’m in a position to offer just a little bit advice.

First, I’ll say that hundred-slide decks annoy me. Yes, I know it’s usually a reaction to the problem of slides that are too few and too densely packed, leading to the also-awful phenomenon of the presenter spending most of the time just repeating what everyone can already read, but it’s an over-reaction. The other day I was reading some slides online, and I encountered the following pattern:

Slide N-1: (clip art)
Slide N: “vs.”
Slide N+1: (more clip art)

A whole slide just for “vs.”? That’s wasting my time. Presenters who use that style end up spending too much of their presentation actually changing slides and waiting the obligatory five seconds for the audience to catch up, no matter how little content is on each. Stephen Foskett pointed out that Lawrence Lessig only puts one word on each slide and is still a very highly regarded speaker. Well, yeah, he’s Lawrence Lessig. I’m not, you’re not, and probably neither is anyone you know (unless of course you know Lessig).

Now, I know presentation length can be tricky. I myself do tend to err on the side of making my slides too busy and very spare graphically. I do that because I know that the slides are likely to be viewed more in email etc. than with me actually presenting them, so to make sure they’re useful as a reference I often sacrifice a little on the “live” side. What I’d generally like to do is create two decks – one verbally spare and graphically rich to illustrate or anchor what I’m saying live, and a longer form for sending around later. That means even more time spent in Impress, though, and is often not feasible for various other reasons as well. My best advice is to determine a good “minutes per slide” figure based on the content, the audience, and an honest appraisal of your own ability to keep the audience interested while the slides aren’t changing, then use that to determine an appropriate slide count. If you’re a very dynamic speaker, you can go the Lessig route and spend five minutes on a one-word slide. If you need a hundred slides to fill a thirty-minute presentation, then maybe you’re admitting something about your speaking skills or the intrinsic value of what you’re presenting.

Second lesson: don’t get too cute. I’ve seen too many presentations lately, especially in the “edgier” tech areas, where the author had obviously spent way more time on finding funny clip art and quotes than on the actual content. Again, it’s a balancing act. Humor is good. A good quote or graphic can be an absolutely fantastic anchor for an important point, which you then elaborate or build on verbally. One not-really-funny slide after another after another with too little in between is just distracting.

Another error that I find even less excusable is simple ugliness. Yesterday I saw a presentation which had been done entirely – from title to closing – in what looked like a version of Comic Sans done to look like paint-brush strokes (house painting, not portrait painting). It wasn’t very readable, and looked totally amateurish. I was embarrassed for the author.

Now, somebody’s probably going to think I’m saying that I’ll totally dismiss an otherwise good presentation of an important idea because of slide count or graphics or font choice. Not so. I’ll still listen, but it will cost the author a “point” in my mind. It’s worth keeping in mind that, in these situations, every single point can matter. If you’re presenting to hundreds of people and only care if one or two respond in any significant way to what you’re saying, then maybe none of this matters. Far more presentations are given in smaller groups, though, where the opinion of everyone at the table does matter. People being how they are, they will use all sorts of nuances to form an impression of whether you’re smart, whether you’re trustworthy, etc. It probably won’t be one big thing that causes you not to get that next meeting, but an accumulation of little things. (If you think “meritocratic” open-source techies are any different, BTW, you’re just kidding yourself. The standards are different, but they’re just as stringently applied. Set the wrong tone and you’ll be written off just as surely and completely.) Why give someone the chance to think that you’re too serious or too frivolous, that your presentation shows disorganization, poor prioritization or disrespect for others’ time or sensibilities? Focus on content, by all means, but take just a little time to make sure it’s being delivered in a way that will ensure a good reception.

Converts, Turncoats, and Moles

Everyone knows that the market for products can be very competitive, and that the competition can often take on a distinctly shady character. What seems to be less appreciated is that the “marketplace of ideas” can be just as competitive, and often just as shady. The battles for “mind share” between one project and another, or even between one approach and another, can be fierce. At the more obviously commercial end of this spectrum are debates like NAS vs. SAN, 10GbE vs. IB, iSCSI vs. FCOE, end-to-end vs. embedded functionality. Often the commercial interests and biases of the participants are obvious, but other times less so. When Joe Random Blogger weighs in on one of these debates, finding out that Joe spent months and thousands of dollars to become a CCIE might shed some light on their vested interest. Other times, there’s no such obvious marker but the vested interest is just as real. This is why Stephen Foskett and others have called for explicit disclosure of such interests by bloggers. Full disclosure: I’ve met Stephen, I like Stephen, we’ve done an Infosmack podcast together and he invited me to drop by at a Tech Field Day where I got to enjoy some free food at EMC’s expense. See how easy that was?

Where things get a little murkier is where there’s not an obvious product involved, or where some of the projects are non-commercial. Some of the participants in the fierce SQL vs. NoSQL debates have a commercial stake, some have a personal stake, and some really have no stake at all beyond a desire to see open discussion advance the state of the art. Quick: which category do I fit into? I’m not even sure myself. I like to think I fit into the last category, but this stuff is not entirely unrelated to the job that puts food on my family’s table so arguments that I belong in one of the other categories have plenty of merit. It’s funny that in my day job I rarely get exposed to such conflicts of interest but as a blogger I do. I’ve been asked to write “objectively” about certain products or projects, or sometimes to refrain from writing about them. I’ve tried to ignore those requests as much as possible, and just write about what interests me . . . but I digress. What are we to make of sniping e.g. between Cassandra and HBase advocates, for example, when both are open source? At an even more abstract level, what about centralized metadata vs. “floating master” vs. peer-to-peer distribution, or using the same vs. separate algorithms for wide-area and local replication? What about public cloud vs. private cloud and the people who claim one or another is beneath contempt? The resolutions of such debates have clear implications for certain projects, and the participants are hardly unaware of that, but the debates are not directly about the projects.

All of this creates an environment ripe for manipulation by the less ethical. Here on this blog I’ve often posted about apparent instances of FUD and astroturf, which are two forms of such manipulation. Bloggers or Twitterers with undisclosed interests in one side of the debate are everywhere. I’ve seen one fellow with a vested interest in certain NoSQL projects repeatedly bash other projects quite savagely, more than once, for faults that his own pet projects still have or had until only a week before, without disclosing his direct involvement in the alternatives that remain after the bashing is done. One of the dark sides of open source is the practice of mining competitors’ code for flaws not so they can be fixed but so that they can be used as ammunition in the war of ideas. Perhaps the most effective technique I’m aware of in this area, though, is the wooing of converts. As much as we all like to pride ourselves on being completely rational and empirical, technology is still a social enterprise and nothing can help one side of a debate more than winning over a prominent member of the other team. “I used to believe in SAN/embedded/centralized but I’ve seen the error of my ways” can be very powerful. It strongly implies that the evolution from novice to expert and that from one position to the other are somehow linked. Novices are fooled into believing X, but experts have figured out Y. Sometimes I’m sure the change in heart is legitimate and sincere, driven by increased knowledge just as it seems or perhaps by the ever-changing tradeoffs we all have to make. (I’ve done this myself, with regard to storage vs. processing networks and the balance of traffic between the two in a distributed filesystem.) Other times, I’m just as sure that someone’s change of heart is the result of deliberate persuasion. Contact might have been deliberately made, perhaps through a mutual friend. The strongest features, situations, and (perhaps non-obvious) future plans/directions for one alternative might have been shared, all deliberately planned but presented as innocent exchange of ideas between colleagues. Sometimes a convert can be won this way, and since nobody is ever as zealous as a new convert the result can often be advocacy of the new preference even in situations where the old preference remains objectively better. Yes, you can buy publicity like that. What you can also do is create “moles” who gain some level of notoriety within a community – it’s really not that hard with emerging technologies – and then very noisily “defect” to an opposing camp.

I don’t know for sure how prevalent this sort of manipulation is. I’ve certainly seen plenty of astroturf and FUD, I’ve seen some of the attempts to persuade “thought leaders” one way or another, but I don’t know for sure if I’ve ever actually seen a mole. What I do know is that I don’t see everything, and I’d be a fool to believe these things don’t happen. The means, motive, and opportunity are all there. There are “social media experts” who are paid – and paid quite well – to do almost exactly what I’ve described; I’m sure not all of them have 100% clean hands. I’ve yet to meet a VP of marketing at a startup who would have any qualms, who would hesitate one second, over paying someone to do these things if they thought that person had the capability. I’m not saying we should all start jumping at shadows, but if you see a prominent advocate of low-cost open-source scale-out solutions suddenly start singing the praises of a vendor who is notoriously opposed to all three features, maybe you should at least consider the possibility that their change of position is something other than a total accident.