Yesterday I did a bunch of experiments on three cloud-computing services. Partly this was just to get some actual hands-on experience with each one. Partly it was to calibrate expectations with respect to I/O performance, for New Job which is approaching quickly. On each one, I set up a small instance using Fedora/CentOS (would have used RHEL except that it incurs extra licensing charges). Then I installed gcc, which wasn’t in any of the base images, compiled iozone, and ran some very basic I/O tests. In each case I tested with both one and four I/O threads. Here are some observations.
- Amazon EC2’s instance-storage performance was pretty bad, at 20-25MB/s for write and 40MB/s for read.
- EBS was even worse for write, but quite good for read – up to 90MB/s with one thread and twice that with four. I’m not really sure how the latter number is even possible (do they have dual GigE interfaces on these things?) but I retested to make sure I was transferring more than the instance memory size etc. Thought: maybe I’m exceeding the instance memory size, but not the system memory size. That’s a bit of a gotcha; I think I’ll do some more testing to explore this idea.
- Rackspace (formerly Mosso) was much better at 100-130MB/s for write and 140-190MB/s for read. This is roughly equivalent to EC2’s instance storage; unfortunately there seems to be no direct equivalent of EBS so storage remains completely instance-private. Running a parallel filesystem across instances might be an interesting experiment.
- Flexiscale performance was very similar to Amazon’s, across the board.
There were also noticeable differences other than performance. Cost wasn’t one of them; prices for similarly configured instances seemed very close for all three. Most of the differences had to do with the administration interfaces.
- Amazon had by far the nicest control panel, with by far the largest selection of images and instance types and configuration options (e.g. “elastic IP addresses”). It was also the only one that seemed decently set up for kicking off a whole bunch of instances all at once.
- Rackspace’s control panel was usable, but with far fewer options. Console access is nice, but I’m not wild about getting email every time a server’s state changes. That would get pretty annoying for serious numbers of servers that are used sporadically. I’m also a little disturbed that my “total disk space” still shows 90GB even after I’ve terminated all of my instances. If there’s an additional step to clean that up so I don’t get billed for the whole month, they’ve sure hidden it well; if there’s not, it shouldn’t be reported the way it is.
- Flexiscale has some more serious usability issues to deal with, starting with registration. They ask for a phone number, but reject any familiar form for a US number (they’re in the UK). Hint 1: you have to use “+1 xxx xxx xxxx” but there’s no help or example to tell you that. Next, you have to “add credit” up front – instead of being billed after-the-fact as with the other two – before you can really do anything. Then, having done that, you’re on the wrong screen to start an instance and it’s not at all clear how to get to the right one. Hint 2: click on the Flexiscale logo at the top left. Attaching an extra disk involved another Adventure-like hunt through the interface, and so on. It was simply unpleasant.
I know some people will say I shouldn’t be using control panels, and should use the API instead. This is clearly an option with Amazon, though it seemed unnecessary for what I was doing given how well the control panel worked. Ditto for Rackspace. Flexiscale’s control panel almost seems designed to push you toward using their API instead, but their documentation for it is lousy – missing information on general session setup, clearly necessary calls not mentioned, etc.
Overall, it looks like Amazon and Rackspace get my approval based on this (admittedly brief and lame) test. With its ability to spawn many instances at once and manage them as a group, and its richness of cloud-oriented images, Amazon seems the most truly cloudy . . . but performance is a bit disappointing. For a smaller number of longer-lived instances that might correspond to a more traditional web-service kind of usage, Rackspace seems to offer significantly better performance at approximately the same price. If Flexiscale’s performance were better – as it should be with their directly SAN-attached storage – or if they had superior functionality then the setup unpleasantness could be overlooked, but as it is I can’t think of any reason I would choose them over Amazon.
Update: tried GoGrid as well. The interface etc. is a little nicer than Rackspace, but still behind Amazon in terms of options. Performance is good (50-70MB/s write, 80-140MB/s read). However, the price for a single-CPU 2GB instance is four times what the others charge for comparable instances. There also seemed to be a glitch with a possibly-timed-out control panel session. I went to delete my instance when I was done, and got an error. Ditto when I tried to look at logs or much of anything else, until I logged out and back in again, and was able to delete the instance immediately. OTOH, kudos for including a compiler in the instance image – the only provider so far to do so.
Thanks for the review, just a quick note to say we have a number of improvements planned with the UI and the storage system, so appreciate the feedback, which matches with our current thoughts.
Regards,
Tony Lucas
CEO
Flexiscale
Glad to hear it, Tony. As I was saying recently, wherever there is disappointment there is also opportunity. I’d be glad to check back and report results whenever those improvements hit.
Thanks for sharing these numbers. I’ve been doing some research on this myself. And have stumbled on some similar results on Amazon’s servers.
However, I can’t help but think that this is due to the ‘virtualized’(Xen) based nature of these servers. From the research I’ve done, I/O appears to be the one thing virtualized environments can’t pool effectively, unlike CPU and memory. (Check this link, ‘To put it in a nutshell : Xen I/O is even worse than I previously thought on a RAID array..’ http://lists.xensource.com/archives/html/xen-users/2008-01/msg00813.html )
It’s my understanding Amazon uses Xen. This other post I found on Amazon’s IO even calls the storage process ‘opaque’ and ‘moody’ ( http://orion.blog.heroku.com/past/2009/7/29/io_performance_on_ebs/ ). And they really just publish CPU, memory and bandwidth ratios, you really can’t be sure how many ’servers’ or what type of apps are operating on the same disks. I have a feeling that’s why its ‘moody’, if your stuck in the same on the same box with a lot of virtualized servers or a heavy IO app that isn’t even yours you will pay the price.
I actually stayed away from Amazon I didn’t see much value from the provider I already use, heard of Rackspace never tried it, never heard of Flexiscale.
Another provider I’ve used is Slicehost, service is excellent and their documentation/GUI is pretty good too. However, a little pricey and pretty much felt they were on par with the provider I’ve been using for some time Linode. For my needs, Linode has been pretty good, they also use Xen and they’ve actually been around for some time even prior to Amazon, and they even publish the number of virtualized servers per host, so a little more forthcoming on how many servers are attempting to access the same set of disks.
Tried ElasticHosts. Decent control panel, prices similar to GoGrid but trailing the pack in performance. I’m beginning to think the two market leaders got that way for a reason, not that such a thing should surprise me.
Could you also try out and add CloudLayer by SoftLayer to the comparison? http://www.softlayer.com/cloudlayer_computing.html
I’d love to, but (unlike their competitors) they don’t seem to provide a way to sign up without calling and talking to an actual sales rep. Hard to believe that someone supposedly supporting the next wave of e-commerce never caught up with the last one, but there it is. Since most of my time to work on this is in the evenings and not during business hours, I haven’t gotten around to accommodating their needs.
Actually, no, I have another answer. They seem to have a bit of a patent problem, a bit of a security problem, and a bit of a business ethics problem. I might never get around to setting up any kind of payment to such an outfit.
Hi Jeff,
Interesting comparison, what about testing us with your method ? Do you use directly iozone figures ?
Our default disk are in RAID 60 so disk performances if you look at the transfer rate only will probably be lower than what you have with Rackspace.
Everybody can have a one month free access during summer, just ask here
http://en.gandi.net/hosting/trial/
Nicolas
Gandi.net
My basic formula is something like this:
iozone -c -e -r 64k -s 2g -i 0 -i 1 -l 4
I use 64KB as a compromise between the usual small (4-8KB) and large (1MB+) sizes. In my experience, real applications often operate on such in-between sizes, or can batch/combine to them. It’s obviously nothing like a comprehensive survey of I/O performance, but it’s usually good enough to get into the right ballpark without much effort.
iozone stats from softlayer cloud instance
Run began: Fri Aug 21 04:24:24 2009
Include close in write timing
Include fsync in write timing
Record Size 64 KB
File size set to 2097152 KB
Command line used: iozone -c -e -r 64k -s 2g -i 0 -i 1 -l 4
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Min process = 4
Max process = 4
Throughput test with 4 processes
Each process writes a 2097152 Kbyte file in 64 Kbyte records
Children see throughput for 4 initial writers = 24216.58 KB/sec
Parent sees throughput for 4 initial writers = 23341.31 KB/sec
Min throughput per process = 5857.41 KB/sec
Max throughput per process = 6278.75 KB/sec
Avg throughput per process = 6054.14 KB/sec
Min xfer = 1956160.00 KB
Children see throughput for 4 rewriters = 15265.15 KB/sec
Parent sees throughput for 4 rewriters = 15185.58 KB/sec
Min throughput per process = 3645.75 KB/sec
Max throughput per process = 3915.61 KB/sec
Avg throughput per process = 3816.29 KB/sec
Min xfer = 1982464.00 KB
Children see throughput for 4 readers = 112634.04 KB/sec
Parent sees throughput for 4 readers = 112614.11 KB/sec
Min throughput per process = 28150.48 KB/sec
Max throughput per process = 28168.05 KB/sec
Avg throughput per process = 28158.51 KB/sec
Min xfer = 2096512.00 KB
Children see throughput for 4 re-readers = 112869.56 KB/sec
Parent sees throughput for 4 re-readers = 112858.69 KB/sec
Min throughput per process = 28199.51 KB/sec
Max throughput per process = 28224.43 KB/sec
Avg throughput per process = 28217.39 KB/sec
Min xfer = 2095488.00 KB
iozone test complete.
More numbers are better. Thanks for running the test, jj.
This isn’t avery good review. Performance is not the main driver for these type of platforms. It is flexibility. Yes EC2 has a great interface and the firefox/plug-in combination in particular if good for creating machines. However, it is expensive and for the cost I would expect more flexibity of the machines.
Elastichost and FlexiScale offer a very different solution in that you can dynamically scale machines which means you can match the resources to demand. The alternative to this is to use a vSphere type of platform which for most companies is prohibitive.
Shreyas
I’m sorry, but saying that examining performance makes this a bad review is idiotic. Yes, cloud computing is about flexibility, but some people do care about performance as well. With equivalent flexibility, as it matters to your application, would you rather use the provider with good performance, or use the one with poor performance and have to deploy twice (or four times) as many instances at corresponding cost? If the review isn’t helpful to you, move on. It’s helpful to others. Where’s your contribution to the state of collective knowledge?
Jeff,
Just wanted to let you know we are going to roll out our new Interface soon. I’d very much appreciate your comments when we roll it out for customer testing.
More details in a blog post at http://blog.flexiscale.com
Regards,
Tony.
iozone output for a 512 MB Slicehost instance running Ubuntu 9.04. Warning: I’ve had wildly varying I/O performance at Slicehost! I’ve been known to move VMs just to get on a less-loaded server…
Iozone: Performance Test of File I/O
Version $Revision: 3.327 $
Compiled for 64 bit mode.
Build: linux-AMD64
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.
Run began: Tue Dec 22 16:33:41 2009
Include close in write timing
Include fsync in write timing
Record Size 64 KB
File size set to 2097152 KB
Command line used: ./iozone -c -e -r 64k -s 2g -i 0 -i 1 -l 4
Output is in Kbytes/sec
Time Resolution = 0.010000 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Min process = 4
Max process = 4
Throughput test with 4 processes
Each process writes a 2097152 Kbyte file in 64 Kbyte records
Children see throughput for 4 initial writers = 51000.68 KB/sec
Parent sees throughput for 4 initial writers = 50115.23 KB/sec
Min throughput per process = 12632.92 KB/sec
Max throughput per process = 12954.18 KB/sec
Avg throughput per process = 12750.17 KB/sec
Min xfer = 2032384.00 KB
Children see throughput for 4 rewriters = 48978.04 KB/sec
Parent sees throughput for 4 rewriters = 48799.35 KB/sec
Min throughput per process = 12199.84 KB/sec
Max throughput per process = 12305.79 KB/sec
Avg throughput per process = 12244.51 KB/sec
Min xfer = 2097152.00 KB
Children see throughput for 4 readers = 60658.46 KB/sec
Parent sees throughput for 4 readers = 60647.32 KB/sec
Min throughput per process = 14346.25 KB/sec
Max throughput per process = 16934.37 KB/sec
Avg throughput per process = 15164.62 KB/sec
Min xfer = 1776640.00 KB
Children see throughput for 4 re-readers = 61622.61 KB/sec
Parent sees throughput for 4 re-readers = 61597.08 KB/sec
Min throughput per process = 14817.91 KB/sec
Max throughput per process = 17020.96 KB/sec
Avg throughput per process = 15405.65 KB/sec
Min xfer = 1826752.00 KB
Those are pretty good numbers, especially for one of the smaller instance types. Thanks! The point about performance variation is a good one as well; Randy Bias pointed out the same thing about Amazon’s EBS not too long ago, so these issues probably bear looking into. What I find really interesting is that the Slicehost numbers seem different than those for the Rackspace Cloud, even though they’re parts of the same company. I think I’ll run some more tests on a same-size RC instance, just so the comparison will really be apples-to-apples.
Hi and thanks for this very good review! Jeff, there are news?