Data Extortion

This is a story about the dark side of moving your stuff into the cloud. It does have a (reasonably) happy ending, but along the way there are some important lessons to be learned about the relationship between cloud users and cloud providers, and how it's possible for people on either side to get burned. There are some bits about contract (and other) law, and customer service, and other things as well, but let's begin with what happened this morning.

Between my reduced hours and the Christmas shutdown, I figure I owe Red Hat about 4.8 hours of work this week. I didn't do it on Monday, so I figured I'd do it this morning. I decided to debug some performance-testing scripts, but since my machines at work are all powered off I figured I'd do it on my cloud machine at Host Virtual. For debugging, I only needed to do short runs - no more than forty seconds or one gigabyte per run, as it turns out. I'd done about a dozen of these when my machine became unresponsive. What gives? I looked around all the usual ways, then logged in to my Host Virtual console to see that my VM was locked with the following message.

i/o abuse from your vm - we are investigating

There are two things wrong here. The less important problem is the premature "abuse" accusation. "Anomalous" would have been fine, "excessive" might even have been OK, but "abuse" is insulting a customer for no good reason. More importantly, locking the VM was a complete overreaction. The tools exist to throttle the I/O from a particular VM instead of shutting it down entirely. I've seen such throttling kick in when testing on other providers many times (more about that in a minute). Even when a shutdown is considered necessary, it's never appropriate to do it without notification. By their own admission they were still investigating, but from my perspective they had already gone beyond investigation to accusation, conviction, and execution.

At this point, I submitted a ticket explaining what I had been doing, and suggesting that their reaction had been premature. If they had just admitted as much, things would have been fine. If they had asked me to reduce my I/O load, I would have. Instead, Customer Disservice Representative par excellence "Mark M" replied saying that I had been affecting other customers and violating their Acceptable Use Policy. Unfortunately, there's nothing to back up that claim. There is no I/O limit specified in their AUP. None. The closest they get is this.

We have determined what constitutes reasonable use of our network for the particular services and products you purchase from us. These standards are based on typical customer use of our network, for similar services and products. It is your obligation to monitor the use of your services and/or server(s) to ensure that there are not unusual spikes and peaks in your bandwidth or disk usage.

In other words, they claim to have some numbers in mind, but won't commit to them in their own AUP. Because those limits aren't specified, even by reference to another document or method of notification, they're legally nonexistent. Even now, nobody at HV has identified a limit that I exceeded, by how much, or for how long. They can't claim any AUP violation without such specifics, and thus they can't claim any right to modify our existing relationship in any way. So, their AUP claim is complete bullshit. What about the "affecting other users" claim?

Well, sorry, but tough cookies for them. Do you know who's responsible for meeting their obligations to other customers? Them. As it happens, I know quite a bit about the problems and technologies involved in providing these kinds of services. In the course of becoming an expert on cloud I/O performance, I've done this same sort of testing on about twenty providers. I've seen the "noisy neighbor" problem from both sides. I've seen my own I/O performance go all over the map because of other users, and I've seen it throttled into the ground supposedly to protect other users from me. I don't love being throttled, but it's an entirely valid response so long as its depth and duration are protective rather than punitive. More importantly, it proves that the technology exists. If HV chooses not to apply it, that's their fault. They can't simultaneously preach about meeting commitments to users while spitting on their commitment to me.

The funny thing is that until now I've been one of HV's biggest boosters. They seemed to be one of the few providers whose I/O performance was marred by neither massive variability nor punitive throttling. Little did I know that their "secret" was to kill VMs arbitrarily when they got in the way. In any case, I expressed my skepticism about their AUP claim, and my dissatisfaction with the lack of notification. That's when "Mark M" really stuck his head up his ass.

if the abuse is ongoing and continued your account will simply be terminated and your server deleted.

What we now have is someone threatening a customer with deletion of data in response to a "violation" that has already been called into question. That's extortion. There is absolutely no situation where it would be appropriate to delete a server while such disagreements are still outstanding, and the fact that Marky Mark regards it as a "simple" matter is appalling.

So I've already moved all my data, and I'll be warning everyone away from this decade's version of Feature Price (widely regarded as the worst web host ever, especially since they also tried to take users' data hostage). No big deal, actually. What's far more important is that this could happen to any user, at any cloud provider. Go take a good look at your own AUP, TOS, or whatever you think spells out the obligations back and forth between you and your cloud provider. How many MB/s may you write, for how long, before they decide you're being "abusive"? Bear in mind that anything left vague might be subject to mind-bending reinterpretation, and anything left out (like HV's I/O limits) might be subject to outright fabrication. What recourse do you have if the provider inappropriately terminates service? Do they admit to any obligation regarding preservation of data while there is an ongoing dispute? I've seen a whole lot of these documents, and all of these things are typically missing. Maybe it's time for someone - users, providers, please not the government - to define minimum standards that cloud providers should meet regarding these sorts of issues. The better providers won't have any problem signing up. The worse ones? Well, I suppose they'll keep on threatening and extorting - and losing - their customers.

By the way, welcome to the new site.

Comments for this blog entry