A note regarding the Voldemort tests. The Python client was created as a proof of concept and I am not sure if it has had any production usage. The Java client has had the most testing (it’s what LinkedIn uses) and the performance numbers from the C++ client seem to be good. Some numbers here:

http://groups.google.com/group/project-voldemort/browse_thread/thread/9dd7ee33305da887/371d23a2afe244c4

Best,
Ismael

As for Cassandra, I think those sites are using Cassandra because it’s a great piece of software. I’ll be the first to admit that these benchmark results are just the tip of the iceberg. In particular, they are specific to virtualized environments and they only test “out of the gate performance” with small datasets. A place like Facebook or Digg is likely to run a bunch of servers natively, which represents a very different performance milieu, and they’re much more concerned with huge datasets. A system like Cassandra that’s well proven at that scale, which also presents a data model and features that they can take advantage of, has huge value. I see the Digg story as a crucial point of validation not just for Cassandra – though I offer my hearty congratulations to that team – but for the entire space of alternative data stores. We’re still in the space where a win for one is a win for all. That will change, but for now let’s enjoy it.

as with any JVM-based app, you need to do about 10k of each op you are benchmarking (per node) to let it JIT things before you start measuring.

I’m not sure if this is unduly affecting the results right now. In the course of running all those read and write tests I would have done far more than 10K requests, and I didn’t see any significant change with a second set of runs following the first.

we found that the GIL has a significant impact on doing threaded testing from Python. prefer multiprocessing.

I didn’t use Python threading for pretty much this reason. The tests were ten separate processes.

Cassandra isn’t designed to handle large blob values; it gives you a column model and expects you to take advantage of that

10K isn’t really that large. It’s at the upper end of what should go into a database row or C/C++ structure, but I’ve seen larger items plenty of times. All it takes is a few long strings/arrays/lists. If there’s a data-model issue here, it’s probably that I’m only using one column. Column-oriented stores like Cassandra can make it possible for an application programmer to use fewer operations to get/set related values, or reduce the amount of data that needs to be transferred for a client-side-serialization approach. Then again, there’s a whole range of other features such as enumerations and extra operations besides get/set/delete that distinguish some stores from others. Keeping results directly comparable often means testing the lowest common denominator, unfortunately, but application developers should keep in mind that there’s more to the story.

10 concurrent clients isn’t a whole lot, for any of these systems. I doubt you can max out the CPU on a single Cassandra node with 10 client threads, even a relatively wimpy EC2 VM (once you fix GC/logging/etc options). (And yes, CPU is almost always the first bottleneck you hit.)

I won’t argue that it is, but I’m far from convinced that it should be. Especially in a virtualized environment, where computation is almost as fast as native but network and disk I/O are markedly slower (even with paravirtualized drivers), I don’t think I’d necessarily expect CPU to be the first bottleneck. How much computation really needs to be done per packet or per block, and if CPU is really the limiting factor then how did Keyspace get higher transaction rates with less CPU? Nonetheless, if/when I get back to this I’ll try running with more threads to see what kind of difference it makes. It might make things better, but I wouldn’t be much more surprised if it made things worse.

10 clients is especially not enough when hitting a cluster of 3 — you’re not seeing throughput go up because you’re going from 100% of ops being local to 1/3, so intra-node latency is killing you.

The client was completely separate, so it was 100% remote either way.

FWIW, I don’t mean to be disparaging toward Cassandra at all. If I were doing this “for real” as the basis for something I was actually putting into production, I’d probably do my detailed comparisons with Cassandra and Voldemort (taking advantage of Cassandra’s richer data model as much as possible). Why not Keyspace? Even though it did well in these tests, I remain leery of a single-floating-master architecture at serious scale, and there seems to be more evidence that Cassandra in particular looks better as the dataset gets larger. I think you guys have done good work, and I don’t want that to go unsaid.

A few notes on Cassandra:
– as with any JVM-based app, the jvm will grab as much memory as you let it. That doesn’t mean it actually needs that much… Cassandra defaults to a 1GB heap but you can certainly get away with less, especially on simple benchmarks.
– Also as with any JVM-based app, you need to do about 10k of each op you are benchmarking (per node) to let it JIT things before you start measuring.
– You’ll easily double performance by setting the log level from DEBUG to INFO (unclear if you actually did this, so mentioning it for completeness)
– The CPU load you’re seeing is from bad default GC options. the defaults will be fixed for 0.4.2 and 0.5, but it’s easy to tweak for 0.4.1: http://mail-archives.apache.org/mod_mbox/incubator-cassandra-user/200910.mbox/%3Ce06563880910222021k6e84262cl912bf80772c1dbb@mail.gmail.com%3E
– we found that the GIL has a significant impact on doing threaded testing from Python. prefer multiprocessing.
– Cassandra isn’t designed to handle large blob values; it gives you a column model and expects you to take advantage of that

Finally, some general notes:
– 10 concurrent clients isn’t a whole lot, for any of these systems. I doubt you can max out the CPU on a single Cassandra node with 10 client threads, even a relatively wimpy EC2 VM (once you fix GC/logging/etc options). (And yes, CPU is almost always the first bottleneck you hit.)
– 10 clients is especially not enough when hitting a cluster of 3 — you’re not seeing throughput go up because you’re going from 100% of ops being local to 1/3, so intra-node latency is killing you.