Evil Timeouts

A few days ago, Pierre Phaneuf revived a discussion about timeouts on a very old article. I’ve spent the last few days at the 2008 Lustre User Group meeting in beautiful Sonoma CA and, since I’m stuck at SFO waiting for my red-eye home (blech blech blech), it seems like a good time to write some more about why I think timeouts are evil.

The Lustre folks are making a bunch of changes to deal with timeout issues on very large systems. For example, they’re going to adaptive instead of fixed timeouts, and introducing a new “Network Request Scheduler” that basically does fair queuing at the Lustre level. Why? Because administrators of large and even medium-sized Lustre systems have gotten sick and tired of logs filled with “RPC timeout” messages and “haven’t heard from X for a while so I’m evicting him” and worse. In essence, Lustre is doing a good job of proving that per-request timeouts don’t scale. NRS will help to prevent outright starvation, but that’s only the most immediate manifestation of the fundamental problem. Let’s say node X sends a message and expects a reply. What happens when 100K other nodes have messages in the queue ahead of X’s, and each one takes more than timeout/100K to execute? Same as now, basically. There are many solutions, but pretty much all of them involve making NRS more complex and increasing protocol overhead without addressing the fundamental problem.

In various systems that I have designed and that have actually worked, I’ve followed a fairly simple rule: there is only one component in the system that can determine anything about the health of another node. Everybody else who sent something to that node will wait indefinitely until they get either a response or an indication from that one subsystem that the other end is dead. That way only the heartbeat subsystem has to be designed – and it does have to be designed carefully – to deal with all of the reliability and scalability issues of determining who’s alive or dead. You don’t have N components in the system each responding in different ways and at different time horizons to a real or perceived node failure. That way inevitably leads to disaster.

Note, however, that decisions and actions other than detecting and responding to a node failure can still be timeout-based. The best example is retransmissions over an unreliable network. It’s entirely possible, and in a large system likely, that a message will get lost and the sender will wait indefinitely while the heartbeat subsystem never sees a problem and thus rightly never sends any node-failure indication to end the wait. It’s entirely legitimate and necessary to retransmit in such a case, though of course the retransmission policy should be considered with an eye toward scalability and the implementation should ideally be invisible to higher software layers.

To pick the obvious example, what might the world be like if Lustre had followed this rule? NRS or something like it might still be necessary for other reasons, but not to prevent the ill effects of prematurely deciding a node is dead merely because it’s too overloaded to answer one request quickly enough. Adaptive timeouts wouldn’t even make sense. Perhaps most significantly, users wouldn’t have to implement failover using a completely separate package that triggers action independently of when Lustre itself detects and responds to RPC failures. That’s a tuning nightmare at best, and it’s highly questionable whether you could ever really trust such a setup not to tie itself up in knots at the most inopportune time (i.e. during heavy load).

Another Blog Neighbor

One of the things we do a lot here in the blogosphere is introduce our friends and family when they start blogging. Another thing we do is shamelessly flog the places we work. I’m pleased that today I can do both. Not long after my mother got a blog, Matt Reilly got one too – Big N Computing. Matt is one of the co-founders of SiCortex, responsible for much of how our systems are designed, and one of the people I’m most likely to be talking about when I mention that I work with a lot of smart people. He’s also very opinionated, and often funny as hell. Most importantly, when it comes to blogging he’s a total noob, so go over and help him get it right, OK? Tell him I sent you.

Ubuntu Hardy Heron on Dell E1505

I finally got around to installing this. Here are the steps involved.

  1. Basic install, reboot, BEEP. As usual, one of my first acts after installing is to blacklist the annoying pcspkr driver.
  2. It used my full 1680×1050 resolution right off the bat, but the non-accelerated driver. Didn’t take me long to find EnvyNG, install it, and run it. Voila! Considering how bad X configuration has always been in the past, this was a pleasant surprise.
  3. Wireless seemed to be working pretty much out of the box (modulo my network-specific configuration), which was another pleasant surprise.
  4. Function keys also worked out of the box too, as did front-panel keys. Sweet.
  5. Suspend/resume seemed to work, albeit slowly (about a minute for the resume). Then I found that my wireless went away. I spent a lot of time figuring out that the b43 driver is a piece of crap. It never survives suspend/resume on its own, so I started writing a script to do it. I eventually had something that would work most of the time, with lots of sleeps and retries and such to work around the various race conditions in b43, only to find that I’d usually lose my connection within a minute anyway. Even when I had a connection, packet-loss rates were awful and performance was abysmal. Much, but not all, of the problem seems to be that this driver refuses to set the transmit power to anything near its full value, so it’s like being at the very edge of my network’s range. I eventually resorted to the tried-and-true combination of ndiswrapper and the Broadcom bcmwl5 NDIS driver, and things are a lot better now.
  6. Firefox 3 is still clearly not done. There’s no excuse for not being able to import a Firefox 2 bookmark file. Apparently somebody did write most of the code, because it gets invoked when you do a version upgrade, but they didn’t hook it up so that it would work for a fresh OS install or any of a hundred utilities that worked with the old format. That’s just a developer being lazy. I downgraded to version 2 until version 3 grows up a bit more.

At this point everything seems to be working, so I’d have to say the install was a success. It’s nice to be up to date again.

My Eee Review

Yes, I bought an Eee. I like my Dell E1505 for desktop-replacement use, and I’ve taken it on quite a few trips without undue bother, but it and its power supply are just a little too bulky to carry around all the time. I don’t take it to and from work, when I do have it at work I don’t take it to meetings unless I specifically plan to use it for a presentation, etc. I decided to get something that was a bit more like a PDA in terms of being able to take it everywhere, but with a real keyboard because without one I’d never use it either. They key to the Eee, in my opinion, is that it’s like a PDA in price, too. Many would-be competitors seem to miss the point when they make physically-similar machines with faster processors and more storage (mostly just so they can run bloaty Vista) and then give it an $800 price tag. I wouldn’t have bought this – yes, I’m using it now – even for $600. That’s just too much to risk for something that doesn’t fit into an established usage category, but for just shy of $400 . . . well, OK, I’ll give it a whirl.

I got the 4GB (701) model, because it’s the lowest-price model that has upgradeable RAM and the bigger batteries. It also has more RAM than 2GB models, and comes with more applications preloaded than will fit on those. It also has a camera, not that I care. It was just the only way to get the other features I wanted, and might even preclude my taking it to some places I might go where they don’t like recording devices. The reaction to the 4GB-and-up models seems overwhelmingly positive, but a significant number of people seem to get frustrated with the limitations of the cheapest versions. I ordered mine from Red Barn, because (a) they had it in stock and (b) I could get it with a nice faux-leather skin. I’m a sucker for skins.

So, how is it? I’m not going to gush with enthusiasm, but so far it has been very nice. Yes, the screen is small. I’ve had to learn all the keyboard commands to maximize windows and get rid of menu bars etc. On the odd-aspect (800×480) screen you really notice how much useful display space those things take up. The keyboard is small too, but I’m generally OK with that; the only thing that consistently trips me up is the one-row-too-high placement of the tilde key. The fact that it’s Linux was actually a plus for me, unlike most people, and not just because I work in a Linux-centric office. The fact that it’s Xandros Linux is more of a problem. Blech, what a dog of a distro. Practically the only thing I had to do on my first day configuration-wise was rip out a bunch of Xandros cruft so that the wireless network connection would come back up properly after a suspend/resume cycle. The fact that the Asus package repositories are so incomplete and limited didn’t make me happy either, but that was also easy to fix. Fortunately, there’s a lot of information out there on user-community sites about how to fix some of the more obvious issues.

The system overall has been stable and has worked well, though. I took it on my recent trip to Michigan as an experiment, to see if it was functional enough to be my sole machine on a trip without risk of finding myself flat-footed on a business trip. All I can really say is that it worked fine. I used it in airports, a little bit on the plane, quite a bit at my brother’s place. I did real work on it, troubleshooting stuff back on the office from the terminal at Logan. For what it is, and it will never be mistaken for a primary system, it is a great little box.

Narrow Vision

I don’t have much time to write more about the whole congestion-control thing right now, but if I hear one more person blather about how the next version of DOCSIS will or won’t solve the problem, then I will fly to wherever there and strangle that person with some optical fiber or twisted pair (their choice). A problem that exists within only one last-mile technology is a problem for the service provider alone, and a solution implemented only within one last-mile technology is no solution at all. When you act as though all the world’s a cable modem, you only show that you don’t really understand the problem.

Sarcasm Pills

This is a great parody of a typical drug ad.
Sarcasma Ad
So . . . do you think I need some? Here’s a local copy in case the image link rots. Hat tip to Blame It On The Voices.

Feet of Sand

Sorry for the lack of posts. All the usual reasons apply, plus I’ve been in Ann Arbor for the last few days. Not much to say about that, except that Boingo airport wi-fi rocks but USeless Airways not so much. I’m using Boingo right now on my Eee – more about that in another post – while I wait for my flight home. Anyhoo, here’s the most recent Amy quote that I can remember. It was after reading a long-ish bedtime story, and she said

One of my feet feels like sand.

Cindy and I both knew immediately what she meant, of course, but it’s a unique phrase. I just love it when she exhibits that kind of creativity in expressing herself. Don’t know a word or phrase for something? Make one up! It’s amazing how often it works.

OK, now *this* is bio-diesel!

I never heard of diesel trees before, but apparently such things really do exist. Imagine being able to grow the fuel you need, right in your own back yard.

Blog Annoyances

Grrr #1: One of my articles here was copied by a splog, with somebody else’s name attached.

Grrr #2: I went to report the splog, since it seems to be a whole domain full. The reporting site had a CAPTCHA with about fifty letters and no instructions about which ones are significant. I made a half-dozen attempts, all of which failed, and then decided I’d entertained the moron webmaster enough.

Grrr #3: I found a link to what I thought would be an interesting article: 7 Pains You Shouldn’t Ignore but it turned out to be one of the most useless pseudo-medical articles I’ve ever read. It might have been useful if it identified seven pains that are somehow distinct from the discomforts of everyday life. Instead I find that if I have any pain in my head, chest, abdomen, back or calves I should run to the ER because I’m about to die. That’s everything medical reporting should not be. It is alarmist and it is not informative. Way to shill for the most expensive segment of the generally venal medical profession, Leanna Skarnulis.

Seriously, don’t these people think before they post garbage on the web? I’m pretty obnoxious sometimes, I’m sure a lot of what I post would seem pretty useless to a lot of people, but at least I don’t just splat stuff out there without even the tiniest hope or care about whether it’s just wasting people’s time like these folks did.