Platypus Reloaded, 23 Apr 2018 20:48:00 -0400Shutting Down This Site<p>Some time in the fairly near future, probably while I'm sitting bored in some hotel room, I'm going to start decommissioning this site. That doesn't mean the content is going away, or that I'll stop writing.. I'd almost like to just pull the plug and walk away to start anew …</p>Mon, 23 Apr 2018 20:48:00,2018-04-23:/2018-04-shutting-down.htmlinternetUnpopular Thoughts About Network Neutrality<p>(1) I support network neutrality, but it's a vague term and I support it according to my own understanding of the technical and social issues. I believe that Network Neutrality(tm) as less knowledgeable people have allowed big content providers to define it is broken, and do not support it …</p>Wed, 29 Nov 2017 17:48:00,2017-11-29:/2017-11-network-neutrality.htmlcommunicationsinternetpoliticsWhy I Like "Like" Buttons<p>As a blogger for (I think) about 18 years, and a BBS/Usenet user for almost as long before that, I have a lot of opinions about how people should interact online. To some extent those opinions are based on solid thinking about how the nature of that interaction shapes …</p>Sat, 08 Apr 2017 16:46:00,2017-04-08:/2017-04-like-button.htmlinternetMoving to Facebook<p>On March 17, I'll be leaving Red Hat. On April 3, I'll be starting at Facebook. I will <em>not</em> be leaving the Gluster project.</p> <p>I've drafted two much longer posts explaining the reasons for this change, but I'll try to keep this one short. It's not a reflection on Red …</p>Sun, 05 Mar 2017 19:31:00,2017-03-05:/2017-03-moving-to-facebook.htmlglusterfsworkingCollecting my thoughts about Torus<p>The other day, CoreOS announced a new distributed storage system called <a href="">Torus</a>. Not too surprisingly, a lot of people have asked for my opinion about it, so I might as well collect some of my thoughts here.</p> <p>First off, let me say that I like the CoreOS team and I …</p>Mon, 06 Jun 2016 13:48:00,2016-06-06:/2016-06-torus.htmlstoragemarketingReal Freedom vs. Illusions of Freedom<p>Randy Bias has taken it upon himself to explain that <a href="">vendor lockin is unavoidable</a>. How surprising that someone who works for one of the most aggressively proprietary vendors in the storage space would say such a thing, with a healthy dose of sneering about unicorns and Santa Claus thrown in …</p>Tue, 10 May 2016 19:47:00,2016-05-10:/2016-05-real-freedom.htmlstoragemarketingUpdating POSIX<p>"POSIX is obsolete." If you're a filesystem developer, you've probably heard that many times. I certainly have. It doesn't tell me anything I didn't already know about POSIX, but it does tell me two things about whoever says it.</p> <ul> <li> <p>They don't know what POSIX is.</p> </li> <li> <p>They're lazy.</p> </li> </ul> <p>To the first …</p>Tue, 26 Apr 2016 10:18:00,2016-04-26:/2016-05-updating-posix.htmlstorageStone Age Programming<p>As a systems programmer, I get to work with a lot of old-fashioned code and tools. The code base I work on every day is in C, complete with manual memory management and constant checking of return values instead of exceptions. Heck, the Gluster coding style even involves using "goto …</p>Thu, 28 Jan 2016 10:06:00,2016-01-28:/2016-01-stone-age-programming.htmldesignworkingFirst Race<p>Today I ran for the 200th time this year, fulfilling a promise I'd made to myself ten months ago. That makes me very happy. To make it extra special, I deliberately (since a couple of weeks ago) scheduled my runs so that #200 would be during the Genesis Battlegreen 5K …</p>Sun, 01 Nov 2015 14:46:00,2015-11-01:/2015-11-first-race.htmlrunningSoftware Will Nibble On Storage<p>There's a theme I keep coming back to lately, of the relationship between hardware and software and how that relates to recent industry developments such as the acquisition of EMC by Dell. It has shown up for me on <a href="">The Register</a>, on <a href="">StorageMojo</a>, and last night on <a href="">Twitter</a>, so it …</p>Tue, 27 Oct 2015 18:52:00,2015-10-27:/2015-10-nibble-on-storage.htmlstorageWinter Running in New England<p>I still consider myself a bit of a running n00b. Several months ago, I was even more of one - so much so that I kept running through one of the worst winters anyone here seems able to remember. Paradoxically, that n00b decision seems to have left me in the position …</p>Fri, 09 Oct 2015 14:31:00,2015-10-09:/2015-10-winter-running.htmlrunningA Year Of Running<p>About a year ago, I started running. I say "about" a year because I don't know the exact date. I know it was early July, so it's not quite a year, but I feel like writing about it now so here goes.</p> <p>A year ago, I knew nothing. I didn't …</p>Sun, 05 Jul 2015 13:13:00,2015-07-05:/2015-07-year-of-running.htmlrunningObject Store File Systems<p>Several years ago, Amazon created something called S3 - Simple Storage Service. The "simple" part was based on the premise that distributed file systems are too complex, inhibiting scalability while providing too little marginal value to users. According to that theory, a system with a simpler API and semantics (e.g …</p>Tue, 23 Jun 2015 08:50:00,2015-06-23:/2015-06-object-store-fs.htmlcephcloudopenstackstorageHTTPS Everywhere and Civil Debate<p>I really don't want to get in the middle of the "HTTPS Everywhere" debate, but a <a href="">recent message</a> on the topic by Roy Fielding (of REST fame) really bothered me, so I'll add my voice to the chorus anyway. Let's start with the non-technical problem with that email, just to …</p>Fri, 05 Jun 2015 15:16:00,2015-06-05:/2015-06-https-everywhere-and-civil-debate.htmlinternetpoliticsDistributed File System SBFAQ<p>There's a lot of hype around distributed file systems and their relatives, such as object stores. Every week, it seems, there's a new project claiming to be the the fastest, most scalable, most robust, most space-efficient distributed file system ever, sweeping all precursors before it. Nine times out of ten …</p>Tue, 26 May 2015 13:42:00,2015-05-26:/2015-05-dfs-sbfaq.htmlglusterfsstorageperformanceStop Calling It Neutrality<p>Usually, when anyone in government tries to do anything about issues of equality or fairness, the techie-libertarian reaction is to complain about "legislating equal outcomes" and invoke the spectre of <a href="">Harrison Bergeron</a> as proof. (Hint: it's fiction!) For some reason, "neutrality" doesn't get the same reaction even though it's a …</p>Fri, 13 Mar 2015 08:48:00,2015-03-13:/2015-03-stop-calling-it-net-neutrality.htmlinternetpoliticsContent From hekafs.org<p>Executive summary: all of that stuff's <a href="">over here</a> now. If you have links to it, just change "" to "" and almost everything should work.</p> <p>A while ago, I got a notice that the domain was about to expire. Even though …</p>Fri, 06 Mar 2015 10:26:00,2015-03-06:/2015-03-content-from-hekafs-org.htmlglusterfsLife on the Server Side<p>Of all the projects I've proposed or worked on for GlusterFS, New Style Replication (NSR) is one of the most ambitious. It has two major goals:</p> <ul> <li> <p>Improved handling of network partitions</p> </li> <li> <p>Improved performance, both normally and during repair</p> </li> </ul> <p>Personally, I consider the improved partition handling to be the more important …</p>Wed, 04 Mar 2015 18:28:00,2015-03-04:/2015-03-life-on-the-server-side.htmlglusterfsstorageperformanceHow Erasure Coding is Not Like Replication<p>Many people think of erasure coding as equivalent to replication, but with better storage utilization. Want to store 100TB of data with two-failure survivability? With replication you'll need 300TB of physical storage; with 8+2 erasure coding you'll need only 125TB. Yeah, sure, there's a performance downside, but from a …</p>Fri, 13 Feb 2015 14:50:00,2015-02-13:/2015-02-ec-vs-replication.htmlstorageNotes on File System Semantics<p>Just some random thoughts from an email I sent recently, plus a bonus SCSI war story.</p> <blockquote> <p>As the PVFS folks said long before I came along, some POSIX requirements are inappropriate for a distributed file system. I agree with that, but not with the object-store folks who claim that the …</p></blockquote>Fri, 09 Jan 2015 09:41:00,2015-01-09:/2015-01-fs-semantics.htmlstorageTechnical Debt vs. Technical Risk<p>One of the most useful metaphors in software engineering is Ward Cunningham's <a href="">technical debt</a>. Definitions and interpretations vary, but technical debt is basically all the stuff you're going to fix later because you were in too much of a hurry to do it right the first time. We all know …</p>Mon, 05 Jan 2015 17:02:00,2015-01-05:/2015-01-technical-risk.htmldesignworkingWhy "DSO" is an Awful Term<p>A recent discussion on the GlusterFS development mailing list got a bit hung up on the issue of what is or is not a "DSO" (Dynamically Shared Object). This is one of a many issues with dynamic linking and dynamic loading that I've seen cause problems before, in large part …</p>Fri, 12 Dec 2014 10:39:00,2014-12-12:/2014-12-dso-terminology.htmloperating-systems"Scale Out" Applies to Interfaces, Too<p>Because of what I do for $dayjob, I hear a lot about "scale out" vs. "scale up" in various contexts. Also because of what I do for $dayjob, I get to read a lot of code. Some of it's new and clean. Some of it's . . . not. That's only partly a …</p>Wed, 03 Dec 2014 16:40:00,2014-12-03:/2014-12-scale-out-interfaces.htmldesignThoughts on Running<p>(...and now for something completely different.)</p> <p>Back in July, I started running. That would not be a particularly notable statement for many people, but most people haven't detested running all their lives and avoided it for thirty years. Instead, I've used stairclimbers and ellipticals for many years, but I've grown …</p>Tue, 21 Oct 2014 12:16:00,2014-10-21:/2014-10-running-thoughts.htmlrunningDistributed Systems Prayer<p>Forgive me, Lord, for I have sinned.</p> <ul> <li> <p>I have written distributed systems in languages prone to race conditions and memory leaks.</p> </li> <li> <p>I have failed to use model checking when I should have.</p> </li> <li> <p>I have failed to use static analysis when I should have.</p> </li> <li> <p>I have failed to write tests that …</p></li></ul>Tue, 21 Oct 2014 10:35:00,2014-10-21:/2014-10-dist-sys-prayer.htmldistributedhumorTen Stages of Technology Familiarity<p>Without further ado...</p> <ol> <li> <p>Never heard of it.</p> </li> <li> <p>Yeah, I hear all the hipsters yammering about it.</p> </li> <li> <p>I checked out the docs and examples once.</p> </li> <li> <p>I used it for a side project.</p> </li> <li> <p>We're using it for some new projects at work.</p> </li> <li> <p>We're using it in production.</p> </li> <li> <p>We're using it in production …</p></li></ol>Wed, 10 Sep 2014 14:45:00,2014-09-10:/2014-09-tech-familiarity.htmlhumorStorage Benchmarking Sins<p>I've written and talked many times about storage benchmarking. Mostly, I've focused on how to run tests and analyze results. This time, I'd like to focus on the parts that come before that - how you set up the system so that you have at least some chance of getting a …</p>Mon, 23 Jun 2014 09:34:00,2014-06-23:/2014-06-benchmark-sins.htmldistributedstorageglusterfsvmwarebenchmarksWannabe of the Month: Skylable<p>Every month or two, someone comes along and claims to be the new Best Thing Ever in distributed file storage. More often than not, it's just another programmer who recently discovered things like consistent hashing and replication, then slapped together another HTTP object store because that's what people nowadays do …</p>Thu, 01 May 2014 12:08:00,2014-05-01:/2014-05-skylable.htmldistributedstorageglusterfscephskylableInktank Acquisition<p>I know a lot of people are going to be asking me about Red Hat's acquisition of Inktank, so I've decided to collect some thoughts on the subject. The very very simple version is that <strong>I'm delighted</strong>. Occasional sniping back and forth notwithstanding, I've always been a huge fan of …</p>Mon, 28 Apr 2014 13:23:00,2014-04-28:/2014-04-inktank-acquisition.htmldistributedstorageglusterfscephNew Style Replication<p>This afternoon, I'll be giving a talk about (among other things) my current project at work - New Style Replication. For those who don't happen to be at Red Hat Summit, here's some information about why, what, how, and so on.</p> <p>First, why. I'm all out of tact and diplomacy right …</p>Wed, 16 Apr 2014 10:24:00,2014-04-16:/2014-04-new-style-replication.htmldistributedstorageglusterfsChange the Axis<p>The other day, I was talking to a colleague about the debate within OpenStack about whether to chase Amazon's AWS (what another colleague called the "failed Eucalyptus strategy") or forge its own path. It reminded me of an idea that was given to me years ago. I can't take credit …</p>Thu, 13 Feb 2014 09:04:00,2014-02-13:/2014-02-change-the-axis.htmlopenstackstrategyData Gravity<p>In the last few days, I had an interesting exchange <a href="">on Twitter</a> about the concept of data gravity. For convenience, I'll include the relevant parts here.</p> <ul> <li> <p><a href="">Mat Ellis</a>: Interesting piece by @mjasay <a href="">link</a> … @randybias is right on the money, data gravity is already a big deal on the cloud</p> </li> <li> <p><a href="">me …</a></p></li></ul>Mon, 10 Feb 2014 08:27:00,2014-02-10:/2014-02-data-gravity.htmldistributedstorageTiers Without Tears<p>A lot of people have asked when GlusterFS is going to have support for tiering or Hierarchical Storage Management, particularly to stage data between SSDs and spinning disks. This is a pretty hot topic for these kinds of systems, and many - e.g. Ceph, HDFS, Swift - have announced upcoming support …</p>Fri, 31 Jan 2014 13:07:00,2014-01-31:/2014-01-data-classification.htmldistributedstorageglusterfsThe World Is Not Flat<p>Way back when I was a young pup, either in college or after that but before I started my career, I got to use an operating system called MTS. That stands for Michigan Terminal System. It was created to run on IBM (and later Amdahl) mainframes, when U of M …</p>Sun, 29 Dec 2013 20:00:00,2013-12-29:/2013-12-world-is-not-flat.htmlstoragedistributedData Extortion<p>This is a story about the dark side of moving your stuff into the cloud. It does have a (reasonably) happy ending, but along the way there are some important lessons to be learned about the relationship between cloud users and cloud providers, and how it's possible for people on …</p>Fri, 27 Dec 2013 21:50:00,2013-12-27:/2013-12-data-extortion.htmlcloudlegalRoll Back or Rock On?<p>For a while now, Kyle Kingsbury has been doing some <a href="">excellent work</a> evaluating the consistency and other properties of various distributed databases. His <a href="">latest target</a> is Redis. Mostly I agree with the points he makes, and that Redis Cluster is subject to inexcusable data loss, but there is one point …</p>Wed, 11 Dec 2013 17:32:00,2013-12-11:/2013-12-roll-back-or-rock-on.htmldistributedGiving Thanks<p>This was inspired both by a <a href="">blog post elsewhere</a> and by a nice email I got this morning thanking me for this blog (thanks Tristan). It seems like we all fail to give thanks, and nowhere more so than in the "gift economy" of open source. I'll start with all …</p>Thu, 28 Nov 2013 11:43:00,2013-11-28:/2013-11-giving-thanks.htmlShared Libraries are Obsolete<p>I was around when shared libraries were still a new thing in the UNIX world. At the time, they seemed like a great idea. On multi-user systems like those I worked on at Encore, static linking meant not only having a separate copy of the same code in every program …</p>Tue, 26 Nov 2013 17:23:00,2013-11-26:/2013-11-shared-libraries.htmloperating-systemsFixing Fsync<p>When I wrote about how <a href="">local filesystems suck</a> a while ago, it sparked a bit of debate. Mostly it was just local-filesystem developers being defensive, but Dave Chinner did make the quite reasonable suggestion that I could help by proposing a better alternative to the fsync problem. I've owed him …</p>Fri, 22 Nov 2013 12:30:00,2013-11-22:/2013-11-fixing-fsync.htmlstoragelinuxThe "IOPS Myth" Myth<p>It's nice to see more people becoming aware that IOPS are not the be-all and end-all of storage performance. Unfortunately, as with all consciousness raising, the newest converts are often the ones that take things too far. Thus, we get extreme claims like <a href="">IOPS Are A Scam</a> or <a href="">Storage Myths …</a></p>Thu, 21 Nov 2013 09:57:00,2013-11-21:/2013-11-iops-myth.htmlstorageperformanceSecure Email<p>Ever since one of the talks at LISA, I've been thinking about secure email. My thoughts are nowhere near complete, but I need to get them out of my head and I do that by writing about them. Apologies in advance.</p> <p>I've actually been thinking for many years about how …</p>Thu, 14 Nov 2013 12:26:00,2013-11-14:/2013-11-secure-email.htmlsecuritycommunicationsMoot Comments<p>I have a couple of posts coming up where I'll be soliciting feedback, so it's time to implement blog comments again. After looking at the alternatives, I eventually decided that <a href="">Moot</a> had the best combination of features for me (as the guy who has to integrate them) and my users …</p>Thu, 14 Nov 2013 10:29:00,2013-11-14:/2013-11-moot-comments.htmlmootcommentsComedic Open Storage<p>I've written before about some people's <a href="">mania</a> for object storage as an alternative to blocks and files. It's a valid model, but I do think its benefits are being pretty drastically oversold. Often there's a lot of FUD about distributed filesystems in particular, from people who clearly don't know the …</p>Thu, 24 Oct 2013 11:32:00,2013-10-24:/2013-10-comedic-open-storage.htmlstorageWhy You Don't Need STONITH<p>(This started as a <a href="">Hacker News discussion</a> about an <a href="">article on Advogato</a>. The articles title/premise is "Why You Need STONITH" where "STONITH" means "Shoot The Other Node In The Head" and is an important concept in old-school HA. I might even have been present when the acronym was coined …</p>Tue, 22 Oct 2013 09:26:00,2013-10-22:/2013-10-stonith.htmlarchitectureLeaning Out<p>In April of '89 I left my family and friends to move from Michigan to Massachusetts for a programming job. The new job paid twice as much as my first programming job had, which means three times as much as I was making since that company laid me off, so …</p>Thu, 10 Oct 2013 20:01:00,2013-10-10:/2013-10-leaning-out.htmlworkingModel Checking<p>Model checking is one of the most effective tools available for reducing the prevalence of bugs in highly concurrent code. Nonetheless, a surprising number of even very smart and very senior software developers and architects seem to know about it. Of the many such people I've worked with over the …</p>Fri, 27 Sep 2013 15:24:00,2013-09-27:/2013-09-model-checking.htmlprocessSAN Stalwarts and Wistful Thinking<p>I've often said that open-source distributed storage solutions such as GlusterFS and Ceph are on the same side in a war against more centralized proprietary solutions, and that we have to finish that war before we start fighting over the spoils. Most recently I said that on Hacker News, in …</p>Wed, 11 Sep 2013 11:38:00,2013-09-11:/2013-09-wistful-thinking.htmlstorageglusterfsmarketingStanding Desks<p>A while ago, I got an <a href="">Ergotron WorkFit-S</a> sit/stand monitor mount. I love it, and have talked about it to plenty of people. Yesterday I joined a <a href="">Hacker News discussion</a> about standing desks, and it left me with some thoughts that I'd rather share here than there, so here …</p>Wed, 28 Aug 2013 09:16:00,2013-08-28:/2013-08-standing-desks.htmlworkingLocal Filesystems Suck<p>Distributed filesystems represent an important use case for local filesystems. Local-filesystem developers can't seem to deal with that. That, in a nutshell, is one of the most annoying things about working on distributed filesystems. Sure, there are lots of fundamental algorithmic problems. Sure, networking stuff can be difficult too. However …</p>Mon, 26 Aug 2013 12:42:00,2013-08-26:/2013-08-local-filesystems-suck.htmlstoragelinuxTechnical Credit<p>To a first approximation, "software engineering" refers to all of the things you need to know when you take "programming" and try to scale it up - more code, more people, more time. You don't need an a civil engineer to dig a latrine, but you'd better have one to design …</p>Fri, 16 Aug 2013 12:06:00,2013-08-16:/2013-08-technical-credit.htmlprocessGlusterFS 3.5 Features<p>It's time to let some cats out of some bags. As my loyal readers (yeah right) have surely noticed, things have been quiet around here. Part of that has been the result of vacations and such, but also there's a lot of stuff I just haven't felt ready to write …</p>Tue, 13 Aug 2013 11:15:00,2013-08-13:/2013-08-glusterfs-35-features.htmlstorageglusterfsAvoiding Jet Lag<p>And now for something completely different...</p> <p>As part of my job - educating and evangelizing and whatever else you call it - I travel a fair amount. I know there are other people who travel ten times as much as I do, but then there are many more who travel less than …</p>Thu, 25 Jul 2013 18:50:00,2013-07-25:/2013-07-avoiding-jet-lag.htmltravelStartups and Patents<p>This should be a <a href="">pretty familiar story</a> to anyone in high tech by now. Startup makes something cool, becomes a target for patent litigation from what we used to call an NPE (Non Practicing Entity). Apparently the new term is PAE (Patent Assertion Entity) but I prefer an even more …</p>Fri, 19 Jul 2013 16:19:00,2013-07-19:/2013-07-startups-and-patents.htmllegalSmall Synchronous Writes<p>Sometimes people ask me why I always use small synchronous writes for my performance comparisons. Surely (they say), there are other kinds of operations that are more common or more important. Yes there are (I say), and don't call me Shirley. But seriously, folks, there are definitely other kinds of …</p>Thu, 18 Jul 2013 10:33:00,2013-07-18:/2013-07-why-sync-writes.htmlstorageperformancePerformance Measurement Pitfalls<p>One of the problems with measuring and comparing performance of scalable systems is that any workload capable of producing meaningful results is going to be highly multi-threaded, and most developers don't know much about how to collect or interpret the results. After all, they hardly ever get any training in …</p>Tue, 09 Jul 2013 20:03:00,2013-07-09:/2013-07-perf-pitfalls.htmlstorageperformanceTwo Weeks is Not a Sprint<p>We're moving to an "agile" development process at work. Yes, we're becoming scrumbags. ;) One of the terms that really bothers me is "sprint" because I think of a sprint as a flat-out effort. That means minimal eating, sleeping, or time with family. Even hard-core hackers rarely do that for two …</p>Tue, 25 Jun 2013 08:10:00,2013-06-25:/2013-06-two-weeks.htmlprocessLies, Damn Lies, and Parallels<p>This apparently happened a while ago, but it recently came to my attention via <a href="">LWN</a> that James Bottomley has made the claim that "Gluster sucks" (not a paraphrase, those seem to be his exact words). Well, I couldn't just let that go by, could I? Why would he say such …</p>Mon, 24 Jun 2013 13:02:00,2013-06-24:/2013-06-lies-damn-lies.htmlstorageglusterfsmarketingPackage Managers<p>There are many things that differentiate a true software engineer from a mere programmer. Most of them are unpleasant - planning releases, reviewing designs or code, testing, release engineering, and so on. One of the most odious tasks is packaging software. I'll admit that it's an area where my self-discipline sometimes …</p>Sat, 22 Jun 2013 12:42:00,2013-06-22:/2013-06-package-managers.htmlprocessMetadata Servers<p>I was sad that I had to miss RICON East, because I knew they had a lot of great speakers lined up. I really liked <a href="">James Hughes's presentation</a>, but must take issue with slide 15.</p> <blockquote> <p>Metadata Servers</p> <p>Required by traditional filesystems (POSIX) to translate names to sectors</p> <p>Hard to scale …</p></blockquote>Fri, 21 Jun 2013 15:47:00,2013-06-21:/2013-06-metadata-servers.htmlstorageStarting Over<p>You might have noticed that things look a bit different around here. OK, if you're reading this in an RSS reader then maybe not, but otherwise it's kind of obvious. I've switched platforms yet again, because I was feeling a bit blocked. Publishing new stuff using my static-wordpress technique was …</p>Thu, 20 Jun 2013 17:38:00,2013-06-20:/2013-06-starting-over.htmlpelican