Thoughts on Fowler’s LMAX Architecture

I have the best readers. One sent me email expressing a hope that I’d write about Martin Fowler’s LMAX Architecture. I’d be glad to. In fact I had already thought of doing so, but the fact that at least one reader has already expressed an interest makes it even more fun. The architecture seems to incorporate three basic ideas.

  • Event sourcing, or the idea of using a sequentially written log as the “system of record” with the written-in-place copy as a cache – an almost direct inversion of roles compared to standard journaling.
  • The “disruptor” data/control structure.
  • Fitting everything in memory.

I don’t really have all that much to say about fitting everything in memory. I’m a storage guy, which almost by definition means I don’t get interested until there’s more data than will fit in memory. Application programmers should IMO strive to use storage only as a system of record, not as an extension of memory or networking (“sending” data from one machine to another through a disk is a pet peeve). If they want to cache storage contents in memory that’s great, and if they can do that intelligently enough to keep their entire working set in memory that’s better still, but if their locality of reference doesn’t allow that then LMAX’s prescription just won’t work for them and that’s that. The main thing that’s interesting about the “fit in memory” part is that it’s a strict prerequisite for the disruptor part. LMAX’s “one writer many readers” rule makes sense because of how cache invalidation and so forth work, but disks don’t work that way so the disruptor’s advantage over queues is lost.

With regard to the disruptor structure, I’ll also keep my comments fairly brief. It seems pretty cool, not that dissimilar to structures I’ve used and seen used elsewhere; some of the interfaces to the SiCortex chip’s built-in interconnect hardware come to mind quickly. I think it’s a mistake to contrast it with Actors or SEDA, though. I see them as complementary, with Actors and SEDA as high-level paradigms and the disruptor as an implementation alternative to the queues they often use internally. The idea of running these other models on top of disruptors doesn’t seem strange at all, and the familiar nature of disruptors doesn’t even make the combination seem all that innovative to me. It’s rather disappointing to see useful ideas dismissed because of a false premise that they’re alternatives to another instead of being complementary.

The really interesting part for me, as a storage guy, is the event-sourcing part. Again, this has some pretty strong antecedents. This time it recalls Seltzer et al’s work on log-structured filesystems, which is even based on may of the same observations e.g. about locality of reference and relative costs of random vs. sequential access. That work’s twenty years old, by the way. Because event sourcing is so similar to log-structured file systems, it runs into some of the same problems. Chief among these is the potentially high cost of reads that aren’t absorbed by the cache, and the complexity involved with pruning no-longer-relevant logs. Having to scan logs to find the most recent copy of a block/record/whatever can be extremely expensive, and building indices carries its own expense in terms of both resources and complexity. It’s not a big issue if your system has very high locality of reference, which time-oriented systems such as LMAX or several others types of systems tend to, but it can be a serious problem in the general case. Similarly, the cleanup problem doesn’t matter if you can simply drop records from the tail or at least stage them to somewhere else, but it’s a big issue for files that need to stay online – with online rather than near-line performance characteristics – indefinitely.

In conclusion, then, the approach Fowler describes seems like a good one if your data characteristics are similar to LMAX’s, but probably not otherwise. Is it an innovative approach? Maybe in some ways. Two out of the three main features seem strongly reminiscent of technology that already existed, and combinations of existing ideas are so common in this business that this particular combination doesn’t seem all that special. On the other hand, there might be more innovation in the low-level details than one would even expect to find in Fowler’s high-level overview. It’s interesting, well worth reading, but I don’t think people who have dealt with high data volumes already will find much inspiration there.

Distributed Databases in 1965

While we were in Ann Arbor last month, we stopped by the abolutely amazing Kaleidoscope used and rare bookstore. (I’d link, but can’t find a website.) I knew from our last visit that they have an excellent collection of old sci-fi magazines, so I decided to see if they had any from the month I was born – April 1965. Sure enough, they had a Galaxy from that month. I was surprised how many of the authors I recognized. Here are the stories mentioned on the cover:

  • “Wasted on the Young” by John Brunner
  • “War Against the Yukks” by Keith Laumer
  • “A Wobble in Wockii Futures” by Gordon R. Dickson
  • “Committee of the Whole” by Frank Herbert

That’s an all-star cast right there. However, the story that really made an impression on me was by someone I had never heard of – “The Decision Makers” by Joseph Green. It’s about an alien-contact specialist sent to decide whether a newly discovered species met relevant definitions of intelligence which would interfere with a planned terraforming operation. That’s pretty standard stuff for the SF of the time, but there’s a twist; the aliens, which are called seals, have a sort of collective intelligence which complicates the protagonist’s job. This leads to the passage that might be of interest to my usual technical audience.

Our group memory is an accumulated mass of knowledge which is impressed on the memory areas of young individuals at birth, at least three such young ones for each memory segment. We are a short-lived race, dying of natural causes after eight of your years. As each individual who carries a share of the memory feels death approaching he transfers his part to a newly born child, and thus the knowledge is transferred from generation to generation, forever.

Try to remember that this was written in 1965, long before the networked computer systems today were even imagined, and that the author wasn’t even writing about computers. He was trying to tell a completely different kind of story; the entire excerpt above could have been omitted entirely without affecting the plot. Nonetheless, he managed to describe a form of what we would now call sharding, with replication and even deliberate re-replication to preserve availability. The result should be instantly recognizable to anyone who has studied modern distributed databases such as Voldemort or Riak or Cassandra. A lot of people think of this stuff as cutting edge, but it’s also an incidental part of a barely-remembered story from 1965. Somehow I find that both humbling and hilarious.

Within Temptation

Last night I went to my first concert in . . . I don’t know how long. I’m pretty sure the last one was before Cindy and I got married, so at least fifteen years. It was quite an experience. For one thing, I’m pretty sure I wasn’t the oldest person there, but I might have been the oldest who was there alone. All the other over-40 types seemed to be escorting kids. That didn’t keep me from being carded, though. The Palladium staff were simply carding everyone who wanted to purchase alcohol, which is totally fine with me.

The Palladium has, to put it bluntly, seen better days. It must have been a grand theatre once, but now things are in pretty serious disrepair. There’s paint and plaster hanging off the walls. Most of the balcony, where I had gone in hope of saving my ears, was closed for what were pretty clearly safety reasons. Somehow the only other unaccompanied 40-year-old in the place happened to be sitting next to me (hi Ben!) and I joked with him a little about the abandoned-theatre ambience providing an appropriate theme for this kind of music. I didn’t catch the name of the opening band at the time, but apparently it was 3. (Side note: naming your band “3″ in the era of Google search seems like a slightly bad idea, because it’s hard to search for.) I liked some of their songs quite a bit, though it also did occur to me that their style might not be the best possible match for the headline act. It was good hard-driving music, with some unique guitar technique adding a bit of flair. After a short break, Within Temptation hit the stage.

What can I say about the music? Of course it was amazing. Everyone has a certain set of musical features that get them excited. For many people, including me for most of my life, this has resulted in combinations that aren’t actually represented by any real-world act. I’m lucky, though, because the combination of mid-tempo high-dynamic range symphonic metal with strong female vocals is pretty well represented in Europe nowadays, and Within Temptation are clear leaders within that genre. Everyone else focuses on Sharon den Adel’s voice, and rightly so, but I think all of the band members play their parts just a bit better than anyone else. Special mention should probably go to Martijn Spierenburg’s work on keyboards, which is often hard to notice consciously but rounds out the sound in an important way. His band-member page also mentions his daughter among his hobbies, which I consider extremely cool. The most important thing about WT’s sound, though, is not the individual contributions but the way they all work together. The buildups are beautifully layered, and the timing when everyone comes in for a crescendo is flawless in both composition and execution. There’s a reason this is the first show I’ve gone to in a long time.

It’s a bit different hearing them in person, of course. First off, it was loud. My ears were still ringing when I got home, and the drive from Worcester to Lexington isn’t short. It wasn’t obnoxiously loud, though. From up in the balcony, at least, the vocals still came through and the overall distortion level was still reasonable so every song sounded just fine – just louder than I’m used to. Maybe that wouldn’t have been true down on the main floor. Also, Sharon’s voice wasn’t 100% last night. She was only twice as good as most other lead singers. ;) High notes were a particular problem, which might have led to disappointment on some of the songs where those particular talents are highlighted. Other than one “well, that didn’t work” shrug near the beginning (which I found endearing) and one not-even-attempted high note later on, though, I didn’t really notice. Like everyone else in the crowd, as far as I could tell, I was just far too enraptured to care about such things. Even at 99%, they provided a musical experience far beyond anything else I can recall. Listening to Within Temptation is a pleasure, but being immersed in the sound like that and with that much energy – from bands and audience alike – is something more.

Yes, I’m a fan. I’m even more of a fan now than I was at this time yesterday, and isn’t that what it’s all about?

Are Unions Leftist?

Yesterday, it being Labor Day and everything, I made a comment expressing appreciation for the gains organized labor has brought. (It turns out that Labor Day might not really be about labor unions so much as the labor – i.e. work – itself, but that’s immaterial here.) My views were immediately branded “leftist” but that seems to be a particularly useless way to consider views toward organized labor. To be sure, unions and strikes have often been used by communists/socialists as weapons against capitalism, and many early union leaders here in the US were very red indeed. On the other hand, some leftist regimes such as China or the old Soviet Union have been markedly intolerant of unions. Go on strike there and you could get yourself shot. Show me a room full of trade-union members here in the US and I’ll show you a room where the dominant political beliefs are those of right-wing Limbaugh, O’Reilly and Beck rather than any “leftist” you could name. These contradictions spring from the fact that, in some essential ways, pro-labor vs. anti-labor sentiment isn’t really a left vs. right thing at all.

Left vs. right has traditionally been about the influence of government in people’s lives, and particularly in their economic lives. The union debate is about tension between two non-governmental centers of power – the providers of capital vs. the providers about labor. Unions are, at the most basic level, just voluntary associations pursuing common goals. The right of free association that’s involved is the same one exercised by the pilgrims at Plymouth Rock or the patriots at Lexington Green. Government has continued to honor this right not only by directly protecting its exercise but also by granting many kinds of associations favorable treatment – especially with respect to taxation. In the particular case of capital holders forming the associations that we call corporations, the benefits extend even further to limited liability, political influence, taxpayer-funded trade diplomacy, etc. If capital providers can voluntarily associate and receive all these benefits for doing so, why not labor providers (i.e. workers)? Why not allow the two groups to work things out directly between themselves, in the market, without government interference in the form of requiring government sanction for unions, enforcing exclusivity (labor monopolies are just as evil as capital monopolies), and so on? How is it “leftist” to recognize that voluntary association in the form of unions is an alternative to direct government interference in the market, and to support that alternative? Support for or opposition to unions is really more of a populist vs. authoritarian issue that doesn’t fit neatly into any simplistic “left vs. right” model. Those who would outlaw unions and replace them with government are leftist authoritarians. Those who would outlaw unions and replace them with nothing at all are neo-feudal authoritarians. I reject both. It’s absurd to think of “power to the people” as a leftist idea . . . but I guess if the right wingers want to make it clear that they’re against that idea then I shouldn’t complain.

[NB: if you go back far enough, to seating arrangements during the French revolution, the identification of "left" with populism and "right" with hereditary aristocracy does seem pretty strong. The axes have shifted since then, though, and by modern definitions the idea of opposing the left by favoring government interference with the right of free assembly seems rather absurd.]

By the way, my support of unions as a concept does not imply support of specific unions as they exist and operate today. Because of government interference such as I’ve mentioned, they have become another form of oppression for individual workers. That’s true whether they’re acting in league with company owners or in opposition to them. Having formerly lived in Detroit I reserve special hatred for the UAW, which in my view has harmed its members more than anyone by steadfastly resisting necessary adaptation to changing technology and globalization. They’ve done as much as the auto companies themselves to destroy that industry and that city. The kind of union I believe in, driven by voluntary participation and popular will of its members instead of being either neutered or captured by the politico-economic elite, hardly exists today. For the sake of real free markets the return of real unions would be a good thing.