Fri 27 December 2013
This is a story about the dark side of moving your stuff into the cloud. It
does have a (reasonably) happy ending, but along the way there are some
important lessons to be learned about the relationship between cloud users and
cloud providers, and how it's possible for people on either side to get
burned. There are some bits about contract (and other) law, and customer
service, and other things as well, but let's begin with what happened this
Between my reduced hours and the Christmas shutdown, I figure I owe Red Hat
about 4.8 hours of work this week. I didn't do it on Monday, so I figured I'd
do it this morning. I decided to debug some performance-testing scripts, but
since my machines at work are all powered off I figured I'd do it on my cloud
machine at Host Virtual. For debugging, I only needed to do short runs - no
more than forty seconds or one gigabyte per run, as it turns out. I'd done
about a dozen of these when my machine became unresponsive. What gives? I
looked around all the usual ways, then logged in to my Host Virtual console to
see that my VM was locked with the following message.
i/o abuse from your vm - we are investigating
There are two things wrong here. The less important problem is the premature
"abuse" accusation. "Anomalous" would have been fine, "excessive" might even
have been OK, but "abuse" is insulting a customer for no good reason. More
importantly, locking the VM was a complete overreaction. The tools exist to
throttle the I/O from a particular VM instead of shutting it down entirely.
I've seen such throttling kick in when testing on other providers many times
(more about that in a minute). Even when a shutdown is considered necessary,
never appropriate to do it without notification. By their own
admission they were still investigating, but from my perspective they had
already gone beyond investigation to accusation, conviction, and execution.
At this point, I submitted a ticket explaining what I had been doing, and
suggesting that their reaction had been premature. If they had just admitted
as much, things would have been fine. If they had asked me to reduce my I/O
load, I would have. Instead, Customer Disservice Representative par excellence
"Mark M" replied saying that I had been affecting other customers and violating
their Acceptable Use Policy. Unfortunately, there's nothing to back up that
claim. There is no I/O limit specified in their AUP. None. The closest they
get is this.
We have determined what constitutes reasonable use of our network for the
particular services and products you purchase from us. These standards are
based on typical customer use of our network, for similar services and
products. It is your obligation to monitor the use of your services and/or
server(s) to ensure that there are not unusual spikes and peaks in your
bandwidth or disk usage.
In other words, they claim to have some numbers in mind, but won't commit to
them in their own AUP. Because those limits aren't specified, even by reference
to another document or method of notification, they're legally nonexistent.
Even now, nobody at HV has identified a limit that I exceeded, by how much, or
for how long. They can't claim any AUP violation without such specifics, and
thus they can't claim any right to modify our existing relationship in any way.
So, their AUP claim is complete bullshit. What about the "affecting other
Well, sorry, but tough cookies for them. Do you know who's responsible for
meeting their obligations to other customers?
Them. As it happens, I know
quite a bit about the problems and technologies involved in providing these
kinds of services. In the course of becoming an expert on cloud I/O
performance, I've done this same sort of testing on about twenty providers.
I've seen the "noisy neighbor" problem from both sides. I've seen my own I/O
performance go all over the map because of other users, and I've seen it
throttled into the ground supposedly to protect other users from me. I don't
love being throttled, but it's an entirely valid response so long as its depth
and duration are protective rather than punitive. More importantly, it proves
that the technology exists. If HV chooses not to apply it, that's their
fault. They can't simultaneously preach about meeting commitments to users
while spitting on their commitment to me.
The funny thing is that until now I've been one of HV's biggest boosters. They
seemed to be one of the few providers whose I/O performance was marred by
neither massive variability nor punitive throttling. Little did I know that
their "secret" was to kill VMs arbitrarily when they got in the way. In any
case, I expressed my skepticism about their AUP claim, and my dissatisfaction
with the lack of notification. That's when "Mark M" really stuck his head up
if the abuse is ongoing and continued your account will simply be terminated
and your server deleted.
What we now have is someone threatening a customer with
deletion of data in
response to a "violation" that has already been called into question. That's
extortion. There is absolutely no situation where it would be appropriate to
delete a server while such disagreements are still outstanding, and the fact
that Marky Mark regards it as a "simple" matter is appalling.
So I've already moved all my data, and I'll be warning everyone away from this
decade's version of Feature Price (widely regarded as the worst web host ever,
especially since they also tried to take users' data hostage). No big deal,
actually. What's far more important is that this could happen
to any user,
at any cloud provider. Go take a good look at your own AUP, TOS, or whatever
you think spells out the obligations back and forth between you and your cloud
provider. How many MB/s may you write, for how long, before they decide you're
being "abusive"? Bear in mind that anything left vague might be subject to
mind-bending reinterpretation, and anything left out (like HV's I/O limits)
might be subject to outright fabrication. What recourse do you have if the
provider inappropriately terminates service? Do they admit to any obligation
regarding preservation of data while there is an ongoing dispute? I've seen a
whole lot of these documents, and all of these things are typically missing.
Maybe it's time for someone - users, providers, please not the government - to
define minimum standards that cloud providers should meet regarding these sorts
of issues. The better providers won't have any problem signing up. The worse
ones? Well, I suppose they'll keep on threatening and extorting - and losing -
By the way, welcome to the new site.