Why Buy RHEL?

Yet again, I’m going to post about something related to my employer. Yet again, I’m going to reiterate that this is not an official Red Hat position. In fact, I more than half expect I’ll get in trouble for saying it, but it just had to be said. You see, there’s a discussion on Slashdot about How Can I Justify Using Red Hat When CentOS Exists? The poster wants the functionality of Red Hat Enterprise Linux, but the CIO doesn’t want to pay for it and demands that they use CentOS instead. A lot of people have tried to explain the various aspects of what a RHEL subscription gets you. I’m not going to expand or correct those comments, because that will definitely get me in trouble and partly because I just don’t care. Here’s the reason that apparently carries no weight at all with CIOs and never even occurs to Slashdotters.

Because it’s the fucking right thing to do, you assholes.

Yeah, I used profanity on what has almost always been a family-friendly blog. I did that because it’s so utterly infuriating that such an obvious and important principle has totally escaped notice elsewhere. If you value something, you pay for it. Even the worst free-market zealots claim to believe that. They often use the same rationale to justify eliminating regulations (especially environmental ones) or replacing public aid with private charity. Red Hat folks do more work than anyone to improve the Linux kernel, GNOME, and dozens of other projects. They write the code, do the testing, fix the bugs, write the documentation, and provide all kinds of logistical support. The beneficiaries include not just obvious derivatives like CentOS and Scientific but even commercial competitors from Oracle and Amazon’s obvious clones to completely separate distributions like Ubuntu which also package that code and fixes. This work isn’t done by volunteers. It costs a lot of money. The fact that we allow the code to be distributed for free should have nothing to do with the principle that you pay for what you value. When you violate that principle you ensure that there will be less of what you value. The result will be a net loss for everyone, as less innovation occurs and more energy is wasted making sure everyone’s “intellectual property” remains under lock and key. Even the thieves lose.

I’d really like to hear from someone who can offer a better moral justification than “we can so we should” for using CentOS on thousands of machines without paying for even one RHEL subscription, because nothing I’ve heard so far is even close. “Duty to maximize profits” arguments will be deleted, because I’ve already turned that one into swiss cheese enough times in my life. Does anybody seriously believe that freeloading should be on the “good” side of our collective moral map?

Libertarian Watch

Alex Tabarrok has written what might very well be the stupidest thing I’ll read this year, about the Mexican Mafia. In it, he portrays their extortion as “taxes” because folks like him love to do the opposite and portray taxes as extortion. He takes it a little further than most, though, by claiming that the MM “became a kind of government” because some of their actions could be construed as protecting property rights or adjudicating disputes. Is that enough to make a government? Is it really equivalent to the torts and courts on which even the most free societies and markets depend? Does the MM provide anything equivalent to national defense – the one institution even the most radical government-haters seem to favor? No, they rely on prison guards, and beyond them the real military, for that. In fact, their whole enterprise depends on Real Government doing all the hard work of delivering victims by incarceration. Tabarrok concludes that the Mexican Mafia has “much to teach us about crime and governance” despite all this. I disagree. An unelected and unaccountable authority defined by ethnic homogeneity and engaging in “taxation” without representation would have no legitimacy as a government, and bears no resemblance to the one with which Alex is not so subtly comparing it. Even a meth habit doesn’t explain that kind of writing.

In other, slightly better, news, Radley Balko has finally figured out that the limited-liability corporation is really an exercise in political economy, and might not be truly compatible with libertarian ideals. Yeah, the “limited liability” part, unaccompanied by anything in return for that governmental favor, kind of gave that away. The corporate structure is to liability what an address in the Caymans is to taxation. Many people have recognized that for years. They’ve suggested that, if we’re going to break the relationship between profit and risk (which real free-market theory tells us is essential), we should at least try to limit or recover the losses that result. Do you suppose that whole careers spent attacking such people as socialist might explain why normal people see “libertarian” as nothing to do with free markets? Of course, the comments to Radley’s article make it quite clear that even asking an innocent question is viewed as heresy. Ours is not to question. Ours is only to accept our position below the New Aristocracy in Washington and Wall Street.

Ganz – Doing Security Wrong

Last week, my mother sent my daughter a gift – a “Mazin Hamster” from Ganz. It comes with a “feature code” that supposedly confers access to a special area of the Webkinz online world. No link; you’ll see why soon enough. The problem is that the hamster’s feature code by itself doesn’t give you access to the Webkinz site. For that, you need the “secret code” associated with a regular Webkinz animal first; then you can use the hamster’s code to get into the special area. Not having such a secret code, I set about procuring one. I went to eBay, found an auction for a cute little gecko with a sealed code attached, and quickly won the auction for far less than it would cost to buy a similar animal in a brick-and-mortar store. So far, so good.

When the gecko arrived, we tried to use its secret code to register on the website. I’m sure everyone can guess what happened next; we were informed that the code had already been used and thus was no longer valid. So here I am, in clear physical possession of both the toy itself and the associated card/ticket with a unique code printed on it, having provably paid for both, but as far as Ganz is concerned I do not own that code. Sometimes possession isn’t nine tenths of the law, after all. The first thing I did was contact the seller, who I will not name because I’m not really sure he did anything wrong. I was polite. I explained the situation, warned him that some of the “sealed” codes on toys he’s selling might not have been sealed in any useful sense after all, and sought his advice. As expected, he swore that the code had been sealed when he got the toy and when he sent it to me. He offered to send me a new code if he got one, but I have to say if I did get a code I could never shake the suspicion that it had come from some other kid’s toy. Having been disappointed twice, Amy was in tears by this point. I don’t much like the idea of merely causing yet another little boy or girl to cry, and I told the seller that.

My next step was to contact Ganz. The phone representative confirmed that the code had already been used, adding that it had been as far back as 2008 and even giving me the first name of who they considered the owner. The toy does appear brand new, in case you were wondering. I’ve seen plenty of these toys before. We even have one (sans code) already, and I can assure you that they don’t stay new-looking long after they get into the hands of a kid who would be interested in registering on the site. Phone Gal also informed me that they do not support sales via Amazon or eBay, only from physical stores or their own eStore. First I’d heard about that. I verified that physical possession of the object didn’t count, and then bade Phone Gal good day.

OK, so I got screwed, but that’s not what this is about. What’s the real problem here? The eBay seller had tried tell me that the codes could be guessed, but I’m skeptical. Each code has to be associated with a particular type of animal. There are enough digits in the code, and enough hundreds of animal types, that making five guesses per day on the Webkinz sites isn’t really going to be very rewarding. No, the first real problem is that the physical security on the authentic codes is very weak. It’s just a simple slip of paper in a plastic envelope tied shut with a little blue ribbon. There’s no plastic thing that you have to break to get at the code, no scratch area, not even a tamper-evident foil seal on the envelope. It would be trivial to buy the toy, use the code, put the code slip back in the envelope, and re-sell it. The physical security is so poor that it would even be possible to do all of this in the store without purchasing anything, and I suspect that’s where most illicitly used codes come from. I was briefly tempted to do exactly that myself, and I’m pretty sure that’s what the eBay seller intended to do, but I try to be a better person than that.

That’s not really the biggest problem here, though. The biggest problem is Ganz’s attitude. They must be aware of how easy it is to steal or misuse codes, and of how often it actually happens. They could secure the codes better, but that might add a couple of pennies to the price. Sadly, I know enough about our collective “race to the bottom” to understand and almost accept that they couldn’t be expected to do that. Alternatively, they could accept proof of physical possession as proof of virtual possession. That would cost them nothing, and would be the fair thing to do according to every moral standard I can think of. Why don’t they? I think it’s because they don’t want to support any kind of re-sale at all. They want to sell you a brand new toy, at full price, even if the toy you already have is only “not new” by virtue of illicit use that they have practically encouraged. Their position is even worse than the RIAA or MPAA, who have at least had to concede that physical transfer of a CD or DVD transfers rights as well. A stolen code is not a lost sale to them; it’s two sales. Doing the right thing would hurt their business. The status quo suits them just fine, and they don’t care how many children’s tears are shed because of it.

No, Ganz, I will not be buying anything from you. Ever. I will endure Amy’s tears if I have to. I will use this as an opportunity to teach her about how companies sometimes do things that are wrong, about the concept of socially responsible purchase decisions, and about boycotts. Then I’ll substitute some other equivalent gift, perhaps a game or membership on some other site, because it’s not her fault (or my mother’s) that you’re evil. I’m so annoyed that I might even do more than that. You’ve made an enemy.

My Brother Rocks

In just about every technical community but one, I probably have a higher profile nowadays than my brother Kevin. I say that not out of younger-sibling competitiveness, but almost for the exact opposite reason – to point out that he’s a pretty technical guy too, and largely responsible for my being one. Here are a couple of points of evidence.

  • His interest in computers predates mine. When a friend of the family loaned us what had to be one of the first TRS-80 computers in New Zealand, it was Kevin who really jumped all over the opportunity.
  • He made a lot more effort regarding computers. One of the very first things he did when he came to the US (a year after our mother and I did) was save up and buy an Apple II. Given the price tag and our economic circumstances at the time, that was a pretty major expenditure. He dove right into 6502 programming, still years before I took programming seriously.
  • He was involved in open-source long before I was . . . except it wasn’t called that back then. Kevin was on the NetHack 3 development team, which was a pretty complex global enterprise. If you were to look at the way the developers coordinated, you’d recognize a lot of the patterns in common use today. This was back in 1989, as I was just starting my own programming career.

Since then, I’ve gone on to infamy and misfortune. Kevin is now a DNS guru, which is why I said “every community but one” earlier. As it happens, this knowledge came in handy just recently. I’m trying to consolidate my web “properties” which are currently spread all over the place. I want to use one provider for DNS, one for email, and one for everything else. GlowHost is very soon going to web-only, and not even that as soon as I get un-stuck enough to set up my own nginx/PHP/etc. configuration on a cloud server I already use for a bunch of other things. As I was trying to move email from GlowHost to FastMail I ran into a glitch. I transferred DNS and email for one of my less-used domains just fine. When I tried to move atyp.us – yes, this domain right here – the DNS part seemed to be OK but I was having trouble with email. I was able to get email on FastMail, but I could see from the headers that it was still going through GlowHost first. I looked at the NS and MX records from a bunch of different places, and everything seemed fine, but even after several days I was still seeing this screwy behavior. Time to call in the DNS expert to see what I was missing.

Pause: can anyone else guess?

The problem turned out to be that mail transfer agents are dumber than I thought, and my silly insistence on using pl.atyp.us instead of atyp.us was confusing the poor babies. Even though I had the MX records for atyp.us and *.atyp.us in place, they’d still fail to find an MX record for pl.atyp.us specifically. Then, they wouldn’t even go “up the tree” and get the MX for atyp.us as I thought they would (and as the SOA for pl.atyp.us makes pretty clear). Instead – and this is the part where Kevin was able to point me in the right direction – they’d fall back to looking for an A record which was still pointing to GlowHost because that’s still where the website is. Bingo. I added the “pl” MX records, and I can already see email flowing in without going through GlowHost.

So thank you, Older Brother. No, not for the MX thing. For every thing.

Stop The Hate

I’ve noticed a significant increase lately in the number of complaints people are making about the operating systems they use, particularly Linux and most especially the storage stack. No, I’m not thinking of a certain foul-mouthed SSD salesman, who has made such kvetching the centerpiece of his Twitter persona. I’m talking about several people I know in the NoSQL/BigData world, who I’ve come to respect as very smart and generally reasonable people, complaining about things like OS caches, virtual memory in general, CPU schedulers, I/O schedulers, and so on. Sometimes the complaints are just developers being developers, which (unfortunately) seems to mean being disrespectful of developers in other specialties. Sometimes the complaints take the form of an unexamined assumption that OS facilities just can’t be trusted, get in the way, and kill performance. The meme seems to be that the way to get better application performance is to get the OS out of the way as much as possible and reinvent half of what it does within your application. That’s wrong. No matter how the complaint is framed, it’s highly likely to reflect more negatively on the complainer rather than the thing they’re complaining about.

Look, folks, operating-system developers can’t read minds. They have to build very complex, very general systems. They set defaults that suit the most common use cases, and they provide knobs to tune for something different. Learn how to use those knobs to tune for your exotic workload, or STFU. Does your code perform well in every possible use on every possible configuration, without tuning? Not so much, huh? I’ve probably seen your developers deliver a very loud “RTFM” when users visit mailing lists or IRC channels looking for help with a “wrong” use or config. I’ve probably seen them say far worse, even. How can the same person do that, and then turn around to complain about an OS they haven’t learned properly, and not be a hypocrite? When you do find those tuning knobs, often after having been told about them because you had already condemned the things they control as broken, don’t try and pass it off as your personal victory over the lameness of operating systems and their developers. You just turned a knob, which was put there by someone else in the hopes that you’d be smart enough to use it before you complained. They did the hard work – not you.

I’m not going to say that all complaints about operating systems are invalid, of course. I still think it’s ridiculous that Linux requires swap space even when there’s plenty of memory, and behaves poorly when it can’t get any. I think the “OOM Killer” is one of the dumbest ideas ever, and the implementation is even worse than the idea. I won’t say that operating-system documentation is all that it should be, either. Still, if you haven’t even tried to find out what you can tune through /proc and /sys and fcntl/setsockopt/*advise, or gone looking in the Documentation subdirectory of your friendly neighborhood kernel tree, or accepted an offer of help from a kernel developer who came to you to help make things better, you’re just in no position to complain or criticize. It’s like complaining that your manual-transmission car stalled, when you never even learned to drive it. Not knowing something doesn’t make you a fool, but complaining instead of asking does. Maybe if you actually engaged with your peers instead of pissing on them, they could help you build better applications.

Gluster Acquisition

For some pretty obvious reasons, everybody’s asking me about this already and will probably continue to do so, so I might as well get some thoughts written down in semi-coherent form. First, though, let’s take care of some administrivia.

I do not – ever – represent Red Hat online. I neither can nor want to speak for them. Also, I was not directly involved in the acquisition. I’m sure my well known opinions about Gluster helped put the idea in people’s heads, and I’m sure I’ll be quite busy helping figure out exactly where to go from here, but it would have been neither appropriate nor useful for me to have been involved in between. Everybody who was involved knew and respected that, as did I, so I was not at all surprised to read about it in public sources first. I’m posting this on my personal site instead of the HekaFS site to underscore the fact that this is my personal, unofficial opinion as someone who is affected by but not responsible for this decision.

OK, enough of that. Personally, then, I am delighted by this. Let’s enumerate some of the things I’ve been feeling and saying about Gluster and GlusterFS since I joined Red Hat and started the CloudFS/HekaFS project.

  • This is an area where open source has needed to make a stronger play vs. proprietary solutions.
  • GlusterFS has a strong overall architecture – e.g. leveraging local filesystems, adding modular functionality – for dealing with emerging needs regarding unstructured data, cloud deployment, etc. Sure, there are some parts of the implementation that I think could improve, but I’d rather build on a strong base than have to rip out and throw away stuff built on a weak one.
  • The Gluster folks are as committed to open source as Red Hat is. Not only their code but their process is open, and so are their minds. Just think for a moment how open-minded someone must be to listen when I get all opinionated about their work. Despite my abrasive style, they have always listened and responded constructively.
  • Community matters, and the very strong Gluster/FS community has been one of the best parts of my job for the last couple of years.

All of this adds up to a move that somebody in open source had to make, and these were the best two companies to make it. Proprietary Big Storage has defined the field too long. This will make it a lot easier to implement not only my vision for HekaFS, but other visions as well. Scale-out shared-nothing storage that’s easy to configure, easy to tune, easy to monitor, is a powerful tool. It can be used to serve up files – or objects – directly to users, as part of either a traditional or cloud environment. It can be used to serve up the virtual-machine (and other disk) images that are an essential part of cloud computing. It can be used for many other things besides, either as-is or via extension modules. Compression or deduplication, snapshots or versioning, custom access controls, inline format conversions . . . the sky’s the limit. Layering separate functionality on top of dumb blocks/files/objects, each oblivious to the other, is so yesterday, but it’s all that competitors will ever make possible. When people have access to a strong and stable core, plus the ability to tinker with it, much more ambitious visions both within and outside Red Hat become possible. What would you do, if you could build storage that was exactly what you need?