Web Mysteries

Do I want to know why weirdsexlaws.com seems to be linking to my site? No, I don’t think so. Even for an infophile like me, some things don’t seem worth knowing.

Flash Memory and Availability

There seems to be a concerted effort among some storage pundits to push the idea that the combination of server-based flash memory devices and commodity-based clustered storage will make disk arrays as we know them obsolete. I’m not sure whether these pundits are deliberately acting on behalf of vendors like Fusion I/O and Ibrix, or whether they’ve merely allowed their opinions to be shaped more subtly by the people they talk to a lot, but the effect is the same. Either way, as always, Robin Harris provides an example. I’m not going to address the issue of clustered storage directly here, having pointed out in a comment that most of those evil disk arrays are already structured that way internally, but I am going to shine a bit of light on server-based flash.

The purveyors of server-based flash – such as Fusion, Violin, and now Micron – make a big deal of the fact that their devices are “closer” to the processor than flash presented via a storage interconnect such as FC (including “tier 0″ usage within a primarily disk-based array). They’re right that such an approach yields higher performance, especially in terms of latency. What they avoid talking about is availability. Being vulnerable to a single-component failure is unacceptable to many people, as is a requirement for physical intervention to restore access to storage – whether it’s disk or flash – that’s “stranded” within a failed server. These are some of the problems that external RAID is supposed to solve, and the parallel filesystems that drive some of the highest I/O loads in the world (e.g. Lustre today, maybe pNFS some day) rely on external RAID to provide certain protections that they do not provide themselves internally.

So, some would say, maybe parallel filesystems should provide protection themselves, via redundancy at the whole-server level. Some do that already, but guess what? If you’re doing that on top of server-based flash, you just threw away your locality advantage. Now you’re accessing it over the same interconnect as you would for disk, which offers better capacity per dollar. Furthermore, in many environments such as HPC, host CPU or memory cycles are not a cheap resource that you can expend or expand more readily than cycles within the storage subsystem. Software RAID, especially in its distributed forms, loses to dedicated RAID hardware in those environments on performance, regardless of the physical medium on which the data reside. (Oops, looked like I talked about storage clustering after all.)

The people who prefer to think of flash as fast disk rather than slow RAM don’t do so because they lack imagination. They (we) often have more experience all up and down the storage/memory and interconnect hierarchies than the people who casually sling that insult. They (we) are not afraid to adopt approaches from historically separate domains, mix approaches, or invent new ones. Usually, their position is based on the simple observation that treating flash more like RAM is at least as fraught with problems as treating it more like disk. It’s really neither, we all know, and server-based flash for data that is private to one server can work really well. If we’re talking about storage that’s shared across a whole set of machines, though, and looking for the best starting point from which to evolve flash into whatever it eventually will become, then I find the in-server proponents’ arguments less than compelling.

BitTorrent over UDP

The hot topic in the blogosphere right now is about BitTorrent Inc. switching to a UDP-based protocol for their popular uTorrent client. Most of the links on this subject ultimately lead back to Bittorrent declares war on VoIP, gamers at El Reg. Having talked to Richard Bennett before, I’m well aware of his penchant for saying outrageous things to get attention, but most of his critics are behaving even worse. For example, here’s Janko Roettgers.

Bennet’s piece is based on a belief that UDP traffic is “aggressive” and uncontrollable, whereas TCP is the nice and proper protocol that can be easily managed. This notion ignores the basic fact that P2P developers, in order to make the protocol work at all, need to implement TCP-like functionalities on top of UDP, one of which includes congestion control. You simply can’t operate a P2P client that eats up all of its users’ bandwidth, much less build a successful business model on top of it.

That’s an unfortunately, not to say selfishly, client-centric perspective. For one thing, eating up all of the user’s own bandwidth is not the issue; crowding out other users’ traffic is. For another, Roettgers completely ignores what happens to packets in between two PCs on the internet. The fact is that all those routers have mechanisms in place to do congestion control, and – while there are proposals for TCP-friendly rate control out there – many people have traditionally turned to UDP for the specific purpose of bypassing congestion control. As George Ou pointed out a while ago, P2P applications tend to “game the system” in effect and probably by intent. I know from my own conversations with Bram Cohen that he has never liked TCP, so it doesn’t really seem all that unlikely that the UDP switch is very intentionally to avoid congestion control – despite the inevitable “collateral damage” to non-BitTorrent users, and therefore quite antisocially. Bennett offers one further reason to believe this.

Upset about Bell Canada’s system for allocating bandwidth fairly among internet users, the developers of the uTorrent P2P application have decided to make the UDP protocol the default transport protocol for file transfers.

I don’t know where Bennett got the Bell Canada angle, and considering the source I’d be interested in any pointers to relevant statements by Cohen or anyone else at BitTorrent, but it certainly seems plausible. What I’d be most interested in, though, is any credible alternative explanation. “It will make BitTorrent downloads faster without affecting anyone else” is just another way of saying there is such a thing as a free lunch. The claims about BitTorrent’s uTP doing congestion control better than TCP are suspect too, since details on uTP don’t seem to be available for us to review and the one detail I’ve seen mentioned – using latency to detect congestion – doesn’t exactly represent the current state of knowledge among people who really understand congestion control. Can anyone explain, without hand-waves and exaggeration, why uTorrent is switching to UDP other than to consume a greater share of bandwidth than existing mechanisms to preserve robustness and fairness in the internet would allow?

Oak Apple Galls

While I’m talking about oak trees, here’s the solution to another mystery. We often find little brown-spotted green spheres in our yard or neighborhood.

Not knowing what they are, I’ve called them pods and Amy has come to consider them an exciting find. As it turns out, “pod” is pretty appropriate because they’re actually galls formed by a species of wasp and even have wasp larvae in them. Click on either the picture above or the one below (from two different sites) for more info.

No Acorns Here Either

One of the disadvantages of having mature oaks in one’s yard is also having lots of acorns to clean up each year. If you leave them, they make already-poor soil even worse, but they’re a pain to pick up. Last year, it seemed like the squirrels were doing a particularly good job; after a couple of years with more than the usual number of acorns, last year was very light. This year we hardly have any. The winter moths took a really heavy toll on our trees this year, so my main theory is that the trees are basically conserving energy by not producing many acorns while the stress persists. The weird wet weather probably had something to do with it too. Apparently there’s a widespread dearth of acorns, with most people attributing it to the standard boom/bust cycles. For a blast from the past, I found an article I wrote in 2004 about the same topic, including a link to a story about very similar conditions leading the the 1968 squirrel migration.