As for sharing more detailed results, I’ve already said I’ll provide them when I’ve run what we all seem to agree are the more meaningful tests. I’m just providing a first look here, not a last word. How are impressions now and details later worse than just details later? How are they worse than nothing now and nothing later, which seems to be what most of the developers for these systems are providing? For example, this is from your own FAQ.

Performance depends on many factors, including hardware and network parameters as well as a great many tunable parameters in the way you set up your cluster and the way that you use Riak from an application. We’ve found it to be fast enough for our purposes, and our goal is not to be “fastest” but rather to stay “fast enough” as the system grows, as hosts fail, and so on. That said, as soon as we get a chance to produce a general, reproducible benchmarking suite, we’ll share it with you.

So, how’s that effort going? Would you like to collaborate on developing such tests? You clearly have some expertise to share, and that would be valuable.

Jeff, I entirely agree with you that measurements that aren’t shared are effectively no measurements. Can you please share your measurements and your benchmarking code? I see no measurements here.

You are also very right that an efficient single-node data store is an essential building block for any distributed data store. However, you’re comparing basic building blocks with entire distributed systems, which is a little confusing to me. Your point is a good one — if you set aside TC’s problems with recovery, it is among the many things whose shape makes it a possible underlying storage container for Riak.

You mentioned that this is an “out of the box” comparison, but Riak’s “out of the box” configuration is optimized for its more common use cases… which are deployments on three or more machines. Among other things, your test was almost certainly writing and reading three replicas of each object on Riak while the other systems were likely only using a single copy. If you only wanted single-system behavior this is easy to set up in Riak, but the default behavior was almost certainly costing a great deal of wasted extra work in this situation.

You said that you used the best available protocols for each system, choosing native interfaces instead of HTTP, etc… but it sounds like you only did that for some systems. It seems that you used Riak’s HTTP interface instead of the native client. While the HTTP interface is very nice and provides some useful features (and excellent interoperability) it certainly isn’t very objective to choose it in this situation. In addition to the protocol marshaling, that interface must do additional computation to produce headers such as Last-Modified and Etag as it goes out of its way to be very well-behaved and cache-friendly HTTP.

Without you providing any of your benchmark code, system specifications, or your output values and statistical methods, it is impossible to have any useful discussion about the “results”. Just saying “8-24x faster” isn’t just apples-to-oranges, it is also entirely vacuous without both data and repeatable methods backing it up.

As for how difficult it is to implement a more robust distributed system, I’m pretty well aware. There are enough bits of my background all over the site to make any convenient assumption to the contrary quite risible. After all, you say you’ve replicated Dynamo. Why do you assume I couldn’t as well, and BTW is your version available anywhere? The 24:1 ratio is just a single observation, not the be-all and end-all of the observations I intend to make or expect others to make. The fact is that an efficient single-node data store is an essential building block for any distributed data store. If one alternative is 24x as fast as another, it’s not at all unreasonable to consider that it might be worthwhile to make it properly distributed despite the difficulty. In this particular case, it might well be possible to combine the upper layer from one of the already-distributed stores with a more efficient single-node store to get something better than either as it is now. As I said in a previous article, scalability is not the same as performance but it’s no excuse for ignoring performance either.

I think we’ll have to agree to disagree about whether or not VMs are good testing grounds. At the root of my dislike is the hypervisor-introduce latency, particularly for disk access; I’ve seen enough in production to make me very leery of using it for low-latency situations. Perhaps that’s acceptable in some use cases.

I’m not fully sure how to respond to your last comment block. Perhaps I should state my biases for starters. I’ve tried using tokyo-cabinent in production and found it to be fast in the short-term and then increasingly slow/latency-unhappy as the amount of data grew. In addition,there were also some significant issues with recovery of TC data in the scenario where the server or host fails; speed (for me) is pointless if you can’t guarantee (most) of my data will be present in the event of a power failure. FWIW, I’m now using embedded Inno and have found that to exceed 2x+ what TC could do in raw requests/sec, while providing decent data integrity.

I’ve also benchmarked both Voldemort and Riak. Yes, V is ~4x as fast as Riak, but it crashes when you cross a threshold of data storage/memory usage. I’ve built my own version of the Dynamo system and have been able to approach V in terms of speed. All this to say that it’s harder, maybe far harder, than you seem to think to build something the provides the sort of data integrity, speed and multi-replica storage on top of Tyrant. So there’s a trade-off there of time to implementation vs. raw perf — that 24:1 number dwindles quickly when faced with the harsh realities of actually implementing a solution. Tyrant isn’t a valid solution for distributed data stores right now, maybe it can be with time. To compare it against existing data stores and say it’s 24 times faster is misleading and unfair, in my opinion. I appreciate your comment re: comparing first impressions with actual experience, but would argue that it’s equally important to ensure that the tests are providing meaningful comparisons.

It’s not my intention to be an ass about this — but perhaps I’m succeeding in spite of myself. I wish more people put the time and effort into benchmarking, and I thank you for the effort.

I respectfully posit that load testing on a VM is something of an oxymoron. You’d never (or at least I wouldn’t) run these sorts of apps in a VM for production purposes due to the unpredictable latency introduced.

I don’t think that’s true at all. These kinds of stores are quite popular in the cloud crowd, where they are often deployed in virtualized environments. Since most of them lack any kind of multi-user security, running them as a non-virtualized service is untenable and anyone who wants to use them in the cloud has little choice but to keep them on-instance behind firewalls.

The warm vs. cold measurements may be misleading due to the caching of the VM and the underlying O/S.

If that were the case, I’d expect the warm vs. cold numbers to end up being similar, but instead they are quite different. More significantly, they are consistently different. Hypervisors actually do less caching than you’d think, because they generally want to devote as much memory as possible to guests which do their own caching.

IMNSHO, latency at some # of requests per second are the two dimensions that are most informative for this type of performance testing.

IMNSHO and IMX that’s true sometimes. Other times it’s requests/second that matters, still other times it’s MB/second. I picked a value and measured it. Maybe I’ll measure another value some time, with different workloads and all sorts of other methodological twists. Gotta start somewhere.

Another big question is how long your load tests ran.

A minute, as I said above. I’m aware that this doesn’t measure effects that kick in when the database is large, and that those can be very important, but it’s also important to measure the starting-line performance and since my time is limited that’s what I did.

What steps did you take to ensure your load generator did not introduce a bottleneck?

Not much. OTOH, there seemed to be little sign of it. I do have some experience measuring both network and storage performance, and I know where to look for excess load. There just didn’t seem to be any on the client side, though there was plenty – CPU utilization, I/O wait time, etc. – visible on the server side in pretty much all cases. If I were trying to measure latency I’d need a more sophisticated test program (part of the reason I didn’t go that route) but for testing throughput multiple instances of a simple test program seems to work fine.

Did you use multiple connections to each server?

Ten, as I said above.

What protocol was used?

Generally “best available” – e.g. native instead of memcached for Tyrant, plain TCP instead of HTTP for some others, etc. I think the quality of the available interfaces is part of the performance equation. Yes, you can do better with some of these by rolling your own hyper-tweaked interface, just as you can do better by using features unique to each, but then the comparisons between them tend to become meaningless. Generally, “out of the box” and “tweaked to the max” are the only configurations worth comparing. This is an “out of the box” comparison and I doubt that I’ll ever have time to compare “tweaked to the max” for more than a couple of alternatives.

Saying that Tyrant is 8-24 times faster than Riak is a bit apples-to-oranges. Tyrant is basically a btree w/ sockets on front. Riak is an eventually-consistent, multiple-replica data store. It’s like comparing a bullet-proof vest to a tank — I know which one I’d prefer to be using when things go south.

Bear in mind that these are single-node tests so they’re more directly comparable than you seem to want. As I discussed above (I see a pattern here), a 24:1 difference in single-node performance strongly implies that a distributed and eventually consistent layer on top of the faster alternative might yield a significntly faster solution than a similar layer on top of the slower one – whether or not such a layer already exists. At that level of difference, one could say to hell with eventual consistency and do synchronous mirroring between a half-dozen Tyrant instances, and still be faster than Riak. I actually kind of wanted Riak to do better, because I like its feature set and what I’ve read about its architecture passes the “how I would have done it” test, but I ran the tests and it fared very poorly even compared to other distributed and eventually consistent stores like Cassandra and Voldemort. I know which one of those I’d prefer to be using when things go south. This is why we do tests, to see whether first impressions survive actual experience, and so far the results seem to suggest that Cassandra might be a better choice. Maybe the results will be different when I run multi-server tests, but I sure as hell wouldn’t assume any such thing.

* I respectfully posit that load testing on a VM is something of an oxymoron. You’d never (or at least I wouldn’t) run these sorts of apps in a VM for production purposes due to the unpredictable latency introduced. Even more so when both VMs are competing for the same hardware.

* The warm vs. cold measurements may be misleading due to the caching of the VM and the underlying O/S. To get an accurate picture (assuming VM testing is acceptable), you’d need to reboot both the VMs AND the host machine to get anything meaningful.

* There are a lot of numbers in your analysis, but no breakdown of actual throughput or latency. IMNSHO, latency at some # of requests per second are the two dimensions that are most informative for this type of performance testing. You also would need a break down of read vs. write requests, since light write load could be significantly faster.

* Without knowing how big your key space (assuming this is all key/value pairs) is, it’s hard to determine how much load was actually exerted on the data stores. Also, using a fixed-size key can be misleading as it permits some structures unfair advantages since they try to match up similar block sizes.

* Another big question is how long your load tests ran. Most data stores slow down as the amount of data grows (typically logarithmically, IIRC). The severity of that curve is tightly coupled to the data structures used. Implementation details matter, a lot, when you start talking more than a few gigabytes of data.

* What steps did you take to ensure your load generator did not introduce a bottleneck? My personal experience is that writing the load generator is as hard, or harder, than writing the original server.

* Did you use multiple connections to each server? What protocol was used? The client protocol (and number of conns) can make a HUGE dent on performance measurements.

* Saying that Tyrant is 8-24 times faster than Riak is a bit apples-to-oranges. Tyrant is basically a btree w/ sockets on front. Riak is an eventually-consistent, multiple-replica data store. It’s like comparing a bullet-proof vest to a tank — I know which one I’d prefer to be using when things go south.