Vacation Email

Maybe this feature already exists, and I never saw it mentioned anywhere. When somebody goes on vacation, they often set a vacation auto-reply to let people know that their message won’t be read until the vacation ends. Some people think this is good etiquette; some think it’s bad. Some employers require it; others forbid it. Anyway, let’s say that you work in a place where a lot of your email is automatically generated, such as by bug trackers or source-control systems. It seems such a waste for my mailer to attempt a vacation auto-reply which will just go into a black hole somewhere. Wouldn’t it be better to prevent such an auto-reply from being generated? Something like this:

X-Allow-Auto-Reply: no

I know there are some headers, like List-Xxx, that can indicate mail is from a list and give a pretty strong hint that an auto-reply would be pointless, but that’s not quite the same thing. Not all auto-generated mail is generated by a list manager or destined for a list, for one thing. For another, I’m sure many people would like to have their mail program insert such a header into every email they send even though they’d be perfectly capable of receiving a reply. Surely I’m not the only person to have noticed this easy-to-eliminate waste, but there doesn’t seem to be any well publicized solution. Can anyone point me to one that might actually be implemented somewhere?

Two Fantasy Novels

In an attempt to use up some of my rapidly dwindling free time, plus some of the proceeds from finally turning my candy-concession change into Amazon credit, I recently bought books by two new fantasy authors – The Name of the Wind by Patrick Rothfuss, and The Blade Itself by Joe Abercrombie. Based on reviews (including comments here) I expected one to be a bit of a curiosity, OK as an alternative to TV or games but ultimately unsatisfying, and the other to be significantly better than that. That’s pretty much how things turned out, but not in the way I expected.

Cloud Services Meetup

I went to my first meetup last night, at Aprigo in Waltham. The group is about cloud services, and on this particular occasion there were two presentations about EMC’s Atmos – one from EMC themselves, and one from partner TwinStrata. Things were pretty quiet at the beginning, as most people were consuming the Intuit-provided pizza and beer. I always think that’s a bit of funny, since it seems to me that the ideal time to eat is when somebody else is talking at the front of the room, and the time when nobody is talking at the front of the room should be prime schmoozing time. Maybe I’m just being too analytical about the whole thing. Anyway, I did get a chance to say hi to Scot Junkin, who I know from SiCortex and is now at EMC, and meet a couple of new people as well.

After a while our host Tsahy Shapsa got things going with a round of quick introductions. There were about sixty people representing the expected mix of entrepreneurs, consultants, and job-seekers. I’m sure some people described themselves as one while actually being another, but that’s OK. That’s what meetups are for, right? Next we started on the EMC presentation about Atmos.

This is Going to be Ugly

This bug allows regular users to put whatever code they want at location zero, and then trick the kernel into executing it. Lovely.

“Since it leads to the kernel executing code at NULL, the vulnerability is as trivial as it can get to exploit,” security researcher Julien Tinnes writes here. “An attacker can just put code in the first page that will get executed with kernel privileges.”

Tinnes and fellow researcher Tavis Ormandy released proof-of-concept code that they said took just a few minutes to adapt from a previous exploit they had. They said all 2.4 and 2.6 version since May 2001 are affected.

Rightly or no, this will blow a big hole in the “Linux is so secure” smugness I often see. Personally, I’ve always thought it was a bad idea to leave user space mapped when you’re in the kernel – for reasons much like this. The model of having completely separate user and kernel/supervisor maps, and special instructions (or instruction variants) to access user space from kernel mode, is less convenient but far more secure.

Cloud Experiments, round 3

I tried running GlusterFS on Rackspace’s cloud, to compare the results with what I’d seen on Amazon. Since Rackspace had shown better raw I/O performance before, I was expecting better filesystem performance as well. That turned out not to be the case, because I ran into a funny cloud-computing pitfall. After I’d dealt with building the code, and dealing with the firewall issues I mentioned before, I did a run and got much lower numbers than I’d expected. Odder still, the I/O seemed very bursty. Watching on one of the two servers, I’d see a burst of I/O every few seconds, with practically nothing in between.

I put this out of my mind temporarily, though, and as I started poking around I realized that I had used the instances’ public IP addresses to connect everything together. Knowing that performance using the internal addresses is usually better, I tried to switch over, but it didn’t work. It turned out that the client could see and connect to server1, but not server2. I double-checked the firewall rules, and everything seemed fine, but there was no connectivity. Not even ping. How strange. Pragmatic guy that I am, I decided to set this mystery aside as well. I rearranged which instances were servers and which instance was the client, so that the node which could see both others (formerly server1) was now the client. Performance was now slightly better, but not much, and still bursty. There was something interesting about the burstiness, though. I/O seemed to alternate between the two servers. Has anybody else guessed where this is going?

At this point I strongly suspect that two of my instances – originally server2/client, later the two servers – were actually on the same physical machine. Thus, they’re not only contending for resources but actually thrashing a bit when I try to run high-throughput tests. I did some more experiments and wasn’t able to prove this theory conclusively, but the evidence seems highly suggestive. This seems a bad VM-placement policy to me. When two instances are started close together (as these were), the optimal policy would most often be to spread them across machines, not to concentrate them on one. Cloud providers, please take note.

Below are some notes, mainly for my own future reference but potentially useful to others, about how to build and run this kind of thing on the Rackspace cloud.

Cloud Experiments, round 2

Over the last few days, I’ve tried to get a couple of parallel filesystems working in various cloud environments. Why? Because I think a parallel filesystem – as we know such beasts today – is an essential building block for the cloud filesystem I’ll be working on. Creating a true cloud filesystem is not a matter of just slapping a new name on an existing parallel not-quite-filesystem. It’s not about minor tweaks to an existing parallel filesystem so that its cluster-developed protocols (barely) work in a more distributed environment. There’s a lot more to it than that, which is precisely why I want to get the parallel-filesystem part settled and move on to working on the other bits ASAP.

Anyway, I rejected Lustre right off the bat because the server side needs kernel patches and this ran up against something I hadn’t realized about cloud services: they make it extremely difficult to run your own kernel. Some simply don’t allow it at all. Others just make the process really cumbersome, and/or don’t provide enough source for you to generate a kernel compatible with the ones they provide – meaning e.g. that you’ll get even worse I/O performance because you have to run even more stuff in fully-virtualized mode. A few don’t even provide enough of the source so you can build your own kernel modules, which has given me a new appreciation for the Affero modifications of the GPL.

I started with Ceph, because it seems in many ways like the coolest of the active projects out there. First I tried to build it on Rackspace. I was in fact able to build it, and install it, on both. Then I spent a lot of time debugging startup failures, which turned out to be caused by the default firewall rules that Rackspace sets up on your instances. I wasn’t terribly pleased with Rackspace for not providing much warning or troubleshooting advice for what I imagine is a pretty common pitfall of using their cloud, or with Ceph for debugging facilities that did very little to shed any light on the matter, but I did get it to work. Unfortunately, performance was extremely poor and it would only stay up a little while before the client would hang. I suspect that it has to do with a couple of Ceph kernel patches that Rackspace surely doesn’t have, so I’m not blaming anyone, but it was a bit of a disappointment nonetheless. For the record, I tried the same on AWS with the same result, except that I didn’t run into the same firewall silliness (this is exactly the kind of area in which AWS seems to have thought things through a little more thoroughly than their competitors).

So, what to try next? I probably should have gone with PVFS, which I still think is the best of its generation. I ended up going with GlusterFS, though, because I knew from past experience that it had the simplest build process of anything I’d found. This time I started on AWS, and got it built pretty quickly . . . or so I thought. It turns out that some FUSE pieces hadn’t really built or installed, silently ignoring the absence of the fuse-devel prerequisite, but that was easy enough to fix. Then I had a heck of a time trying to get a mount to work properly. The documentation mentions two or three different formulae for the client mount command, but they’re all just a little bit wrong. Even when I finally hit on the right general invocation, it kept trying to mount on /mnt even though I’d told it to use /mnt/gfs-data. Eventually I managed to fix that too, and run some reasonable-scale tests without any stability problems.

The results are not bad. With only two servers, and four I/O threads on one client, I was able to get up to 70MB/s write and 90MB/s read. These numbers aren’t directly comparable to my previous numbers since I was using large (64-bit) instances this time, but they’re really not too bad for an untuned configuration using virtualized Ethernet with a 1500-byte MTU. I think I’ll spin up a couple more instances to check scaling, and maybe try the same experiment on Rackspace to get some more data points, but it’s a good start. I’ll probably be posting some more results after my official job start on Monday.

Not Dead Yet

Saying “X is dead” is the new way to put on jaded-veteran airs so you can look down at all those people who don’t know it’s dead yet. Information technology is dead. Operating systems are dead. Feh. Let’s take a look at the “IT is dead” claim, as recently made by Thomas M. Siebel in the NYT, first.

In Mr. Siebel’s view, I.T. is a mature industry that will grow no faster than the larger economy. He contends that its glory days are past — long past, having ended in 2000.

I have no doubt that IT as Siebel knows it is starting to decline, but that only reflects how much IT has continued to evolve while he has apparently stood still. Siebel is a typewriter manufacturer. Back in the day, typing was something that typists and secretaries did. When computers came along, everyone started typing. I’m sure a typewriter of that time might have looked at the declining number of typists and thought “typing is dead” but that would have been a mistake. Don’t confuse the decline of a market with the decline of an underlying technology or activity. In fact, more people are typing now than ever. There’s even a lot of money to be made selling keyboards, though they’re computer keyboards instead of typewriters.

So it is with IT. People are still doing IT, but more people are doing it in more places. You no longer need dedicated equipment or personnel to handle even the simplest computing tasks. Users themselves are installing software, updating web pages, etc. Then there’s that person in every office who does slightly less routine kinds of IT tasks like troubleshooting network problems. Routers and even low-end storage are now sold as office supplies, not as specialized computing equipment. Your DVD player has processors running software, but I’ll bet it doesn’t show up on “IT equipment” sales. My iPod Touch has more computing power than things that used to be called supercomputers, but it’s not “IT equipment” either. A lot of this spending doesn’t show up as IT spending, so people like Siebel don’t figure it into their calculations, but that’s just because they’re counting the wrong thing.

There’s a similar dynamic at work with the “operating systems are dead” meme, which has actually been going around for a while. I’m going to pick an example from Paul Maritz, as quoted by Storagezilla.

Paul Maritz pointed out that today no one really writes to an OS anymore, they write to 3rd generation Frameworks. And he’s right, if we look around we see lots of 3rd generation Frameworks and some of them are more cross platform than others. We all know about Java and .Net but Cocoa for MacOS X (Found on your Mac and your iPhone) Adobe Air, Microsoft Silverlight and so on are all examples of Third Generation Frameworks.

Yes, there are a lot of people working on such frameworks, and even more using them. That’s nothing like “nobody writes to an OS any more” though. Who wants to run any of the following on Air or Silverlight?

  • Development tools, starting with compilers and interpreters.
  • Databases.
  • Web servers (on which many frameworks themselves depend).
  • Email servers.
  • LDAP servers, VPN gateways, etc.
  • Anything in HPC.

The answer is nobody. I don’t think the people behind gcc or Oracle or Exchange are planning to run on top of any such “third generation” frameworks (and BTW I wonder how the third generation is different). They plan to support such frameworks, but that’s different. Operating systems provide a general and stable interface, suitable for many kinds of applications beyond those for which any given framework is likely to be appropriate, and with full performance for when that matters

What might be true is that, the division between “back end” platform (i.e. OS) specialists and “front end” interface or domain-specific specialists will change. Instead of every application-development group being about evenly split between the two, there might be some people developing a few (or a few dozen) frameworks and a much larger number using those frameworks. Same as it ever was. Having been an OS developer for so long, I’m well aware that my kind have long been outnumbered by people working at a much higher level of abstraction who barely understand what I do. Maybe framework developers feel the same way, or soon will. That doesn’t mean OSes are dead, though. People still need device drivers and filesystems and network stacks and schedulers and virtual memory and all that other stuff. OS development will continue, and OSes will continue to compete to capture developers’ hearts and minds. A significant number of people will be watching, and using the results directly to improve old frameworks or create new ones. In fact, the prevalence of framework-oriented workloads is causing a near-term increase in the amount of work that OS folks must do to accommodate those workloads. That doesn’t seem very dead to me.

IT is not dead. OSes are not dead. They’ve evolving. Those who get left behind might find themselves alone and think the party’s over, but in reality they just missed it.

Azure on the Move

Microsoft is moving its Azure operations from Washington to Texas.

the company informed early users of the Azure platform that they’ll need to move their applications off the data center located in Quincy, Washington. Microsoft said it will soon offer users an automated tool that will help them migrate to its Southwest data center, most likely a reference to one in San Antonio, Texas.

“This change is in preparation for our migration out of the northwest region,” the blog post reads. “Due to a change in local tax laws, we’ve decided to migrate Windows Azure applications out of our northwest data center prior to our commercial launch this November.”

…except that it’s not really a change in tax policy.

The relevant tax issue began to unfold in late 2007 when Washington’s attorney general wrote an opinion stating that data centers do not qualify as manufacturers, which in some areas of the state are exempt from paying taxes on new projects, including buildings and the equipment in them. Manufacturers in rural areas also get new employee business and occupation tax credits. Before that opinion was issued, some data center operators believed that they could be considered manufacturers and thus get those tax breaks.

After the attorney general’s opinion made it clear that the data center operators would have to pay those taxes, Washington Governor Christine Gregoire drafted a bill that would have handed data center operators a discount on their taxes for equipment in the facilities, but the bill did not pass.

In other words, this is merely enforcement of an existing policy. It was Gregoire who sought a change in policy, but failed to get it. The sad thing is that the reporting is about Microsoft choosing to move. The story should have been about Microsoft trying to take advantage of a tax break intended for manufacturers, but the “race to the bottom” practice of states and municipalities giving tax breaks and free infrastructure to big companies (which is very rarely justified in terms of either jobs or revenue) has become so commonplace that nobody even notices any more. Free market, my ass. This story couldn’t even happen in a free market.

Thoughts on Health Care

A lot of the debate about health care seems to center around cost and payment. I’m not going to discount the importance of these issues, but I can’t help but feeling we’ve skipped a step. With any policy, before we start considering cost, I think we should consider what is the right thing to do. I happen to think universal, portable health care is the right thing to do both morally (to avoid the de facto rationing/denial of care based on income and employment status) and economically (to avoid the much higher cost of ER visits and other late-stage care vs. prevention and early detection/treatment). Does someone out there not agree? That’s fine. That’s great. Let’s have that discussion. Make your case for why we shouldn’t try to improve the way health care is funded and provided in this country. (Make sure you’ve read Betrayal of Trust first, though, unless you like putting your ignorance on display.) As near as I can tell, the people who are against health care reform seem to have skipped this step in favor of talking about cost first – when they’re not busing hordes of brownshirts to disrupt town meetings, that is.

Let’s say, for the sake of argument, that we’ve determined to reform health care. Now the issue of cost comes into play. In an ideal world, reforming health care would actually reduce costs – e.g. by replacing late-stage intervention with early-stage, by reducing paperwork or duplication of administrative effort – so that nobody would have to pay more. Unfortunately, we don’t live in such an ideal world. Too many are uninsured now, and getting them insured will cost more than we can hope to save. If total medical cost is going to increase, somebody has to pay more than they do currently. Who?

This brings me to my other major point: the opposition to the “redistribution of wealth” implied by increasing taxes on the wealthy (more precisely the high-income). Everybody’s against redistribution of wealth . . . when it suits them. All too often, they’re blind to other kinds of redistribution from which they have benefited. Here’s a hint: any time government money to anyone other than the exact same people from which that money came (which would be pointless), redistribution has occurred. Anyone who has received a student loan, farm subsidy, or government contract has been on the receiving end of redistributed wealth. Where was their outrage then? Nowhere in evidence, of course. Redistribution toward me is the Natural Order of Things; redistribution away from me is the Worst Kind of Socialism. It’s kind of like everyone attributing their success to personal merit and their failures to others’ malice or discrimination, and my response is the same: bollocks. Whether or not a particular kind of redistribution is justifiable has nothing to do with who’s considering it. In the particular case of health care, those who are most secure in their insurance status – notably those on Medicare – have been putting off this crisis by voting for measures (or people who will vote for measures) that will minimize effect on their own costs by maximizing the number of Other People who are uninsured. It’s like a Ponzi scheme, and like all Ponzi schemes it’s not sustainable. The people who have been benefiting from health-care injustice for years are finally being asked to pay in, and now they’re suddenly hyper-aware of the the tradeoffs that they themselves have made inevitable.

The Medicare example makes this even more clear. Baby Boomers on Medicare are one of the major factors driving up overall health-care costs. As a group, they’re relatively wealthy, yet they expect the government to pay for all manner of expensive surgery and medication to correct for their own entirely voluntary lifestyle choices – poor diet, poor exercise habits, etc. – in the past. They love their socialized medicine, so long as nobody talks about means testing, and at the same time they are one of the groups most opposed to providing the same for others. All for me, none for thee. I’m OK, screw you. This attitude is the hallmark of health-care-reform opponents. They don’t care about what’s right. They don’t care about society-wide tradeoffs or long-term effects. All they care about is the immediate effect on their own pocketbook. Sure, they’ll enlist those gullible fools who are willing to believe that their own medical risks are low and their future incomes are high, getting some to forego actual productive work in favor of participation in the town-meeting astroturf campaign, but the people actually driving the anti-reform campaign are doing so entirely out of selfishness.

Go ahead. Prove me wrong, if you think you can. Tell me what your real factually and logically consistent reasons are. I’d love to hear someone even try. Just once.

Amazon vs. Rackspace vs. Flexiscale

Yesterday I did a bunch of experiments on three cloud-computing services. Partly this was just to get some actual hands-on experience with each one. Partly it was to calibrate expectations with respect to I/O performance, for New Job which is approaching quickly. On each one, I set up a small instance using Fedora/CentOS (would have used RHEL except that it incurs extra licensing charges). Then I installed gcc, which wasn’t in any of the base images, compiled iozone, and ran some very basic I/O tests. In each case I tested with both one and four I/O threads. Here are some observations.

  • Amazon EC2′s instance-storage performance was pretty bad, at 20-25MB/s for write and 40MB/s for read.
  • EBS was even worse for write, but quite good for read – up to 90MB/s with one thread and twice that with four. I’m not really sure how the latter number is even possible (do they have dual GigE interfaces on these things?) but I retested to make sure I was transferring more than the instance memory size etc. Thought: maybe I’m exceeding the instance memory size, but not the system memory size. That’s a bit of a gotcha; I think I’ll do some more testing to explore this idea.
  • Rackspace (formerly Mosso) was much better at 100-130MB/s for write and 140-190MB/s for read. This is roughly equivalent to EC2′s instance storage; unfortunately there seems to be no direct equivalent of EBS so storage remains completely instance-private. Running a parallel filesystem across instances might be an interesting experiment.
  • Flexiscale performance was very similar to Amazon’s, across the board.

There were also noticeable differences other than performance. Cost wasn’t one of them; prices for similarly configured instances seemed very close for all three. Most of the differences had to do with the administration interfaces.

  • Amazon had by far the nicest control panel, with by far the largest selection of images and instance types and configuration options (e.g. “elastic IP addresses”). It was also the only one that seemed decently set up for kicking off a whole bunch of instances all at once.
  • Rackspace’s control panel was usable, but with far fewer options. Console access is nice, but I’m not wild about getting email every time a server’s state changes. That would get pretty annoying for serious numbers of servers that are used sporadically. I’m also a little disturbed that my “total disk space” still shows 90GB even after I’ve terminated all of my instances. If there’s an additional step to clean that up so I don’t get billed for the whole month, they’ve sure hidden it well; if there’s not, it shouldn’t be reported the way it is.
  • Flexiscale has some more serious usability issues to deal with, starting with registration. They ask for a phone number, but reject any familiar form for a US number (they’re in the UK). Hint 1: you have to use “+1 xxx xxx xxxx” but there’s no help or example to tell you that. Next, you have to “add credit” up front – instead of being billed after-the-fact as with the other two – before you can really do anything. Then, having done that, you’re on the wrong screen to start an instance and it’s not at all clear how to get to the right one. Hint 2: click on the Flexiscale logo at the top left. Attaching an extra disk involved another Adventure-like hunt through the interface, and so on. It was simply unpleasant.

I know some people will say I shouldn’t be using control panels, and should use the API instead. This is clearly an option with Amazon, though it seemed unnecessary for what I was doing given how well the control panel worked. Ditto for Rackspace. Flexiscale’s control panel almost seems designed to push you toward using their API instead, but their documentation for it is lousy – missing information on general session setup, clearly necessary calls not mentioned, etc.

Overall, it looks like Amazon and Rackspace get my approval based on this (admittedly brief and lame) test. With its ability to spawn many instances at once and manage them as a group, and its richness of cloud-oriented images, Amazon seems the most truly cloudy . . . but performance is a bit disappointing. For a smaller number of longer-lived instances that might correspond to a more traditional web-service kind of usage, Rackspace seems to offer significantly better performance at approximately the same price. If Flexiscale’s performance were better – as it should be with their directly SAN-attached storage – or if they had superior functionality then the setup unpleasantness could be overlooked, but as it is I can’t think of any reason I would choose them over Amazon.