Amy Pix

Some from gymnastics, some from our trip to New Jersey. Without further ado…

Nomadic Platypus

I’ve written before about how good web hosts never last forever, but I didn’t expect that Site5 would decline so rapidly. When I switched to them, a mere three months ago, they seemed to be one of the most respected hosts on some of the sites where people talk about hosts. Now, their reputation is dirt. Thread after thread on WebHostingTalk is about their performance and reliability problems, their lack of response to support tickets, their near disappearance from WHT itself (where they used to participate actively), etc. One thing I haven’t seen mentioned, but which I find annoying, is that my site statistics have gone all wonky. Partly it’s because I seem to be getting many more hits from worms and zombies than had previously been the case, partly it’s because Site5 doesn’t update the stats nightly like I’ve been used to (probably another side effect of servers being overloaded), and partly it’s just random weirdness. The thing that really tipped me over the edge, though, was finding out on WHT that Yahoo had blacklisted Site5 for spamming. So that’s why several of my recent emails never got replies. I’m sorry, but that’s just not something I’m prepared to tolerate. I guess I just managed to catch Site5 at the top of their trajectory, just before their (at the time well deserved) reputation for excellence led to a surge in users, overload, employee burnout, and collapse.

NameZero, JTLNet, FeaturePrice, Burton, TotalChoice, Site5, … now eMax. Some of the content here has lived at seven web hosts, through as many content-management systems (flat files, two home-grown codebases, pMachine, WordPress 1.2 and 1.5 with 2.0 to come soon). You really shouldn’t see any difference other than the little “host du jour” at the top right will flip from “Site5″ to “eMax” at some point over the next few days. If you do notice a difference, please let me know. I’ll still be able to see email and comments at both sites for a while now, because I know all of the magic incantations involved in switching hosts pretty well by now. I don’t expect any host to last forever, but my challenge is to see whether they can outdo the approximately two year run that TotalChoice had.

Funny Amy Story

Last night at dinner, I sneezed. Thinking that Amy might have heard “gesundheit” enough times to come up with a reasonable facsimile, I asked her, “What do you say when someone sneezes?” Her answer, said loudly, clearly, and with obvious pride:


Ahem. Yes. With perfect hindsight, that is exactly what she learned to say when she recently had a cold. Right you are, kid.

Exploding Net-Neut Myths, part 2

Probably the single most common argument used by the net-neutrality folks goes something like this.

I paid for my network bandwidth. Google paid for theirs. Why should either of us get charged again when I access their site?

Like the “double taxation” canard that estate-tax opponents use, this one is more smoke than substance. First, you didn’t pay for your bandwidth. You paid for a physical connection that has a particular maximum bandwidth capability, and for a share of the bandwidth available to those who got the same service you did. If every subscriber to your provider in your neighborhood tried to use their maximum bandwidth at the same time, it simply wouldn’t happen. Networks simply aren’t built that way. If they were they’d be overbuilt, at great expense, for 99.999% of the time, and the people who make networks can’t afford to do things that way. You compete with your neighbors for the bandwidth that is available; you always have and you always will. Check your ISP contract if you don’t believe me. If you want guaranteed bandwidth, you have to pay more for a “service level agreement” with your bandwidth provider, and that’s exactly what the Markey amendment at the core of this debate would preclude.

Second, neither you nor Google paid for bandwidth all the way between you and them. There’s a handy little program called “traceroute” (or “tracert” on Windows) that can show you the path that your packets will take to and from some particular site. Use it. You will see that there are almost always multiple networks involved – yours, Google’s, and not one but several in between. You paid for part of that, and Google paid for part of that, but nobody in particular paid for the rest. What happens instead is that networks set up “peering agreements” in which they agree to transport packets for one another, in what amounts to a “gentlemen’s agreement” that spreads the costs out among all of the so-called backbone providers.

Unfortunately, not everyone’s a gentleman. Some people might remember hearing about peering agreements a few years ago, and it was probably because some networks weren’t being very gentlemanly. They were dumping more of their packets onto other networks than they were accepting in return, sometimes deliberately offloading traffic that they could have carried all the way from their source to their destination by themselves. This shifted costs from them onto others, and others resented it so they started refusing or terminating peer agreements that they felt were inequitable. Imagine for a moment if airlines worked this way, agreeing to accept each others’ passengers when equipment failures and such might otherwise have left passengers stranded. Now imagine that Delta was found to be systematically dumping their passengers on Northwest, even when there was no real need, to save costs. Don’t you think Northwest might have a right to feel aggrieved? Yes, of course they would. One likely solution would be a demand from Northwest that Delta reimburse them for carrying the dumped passengers – but, again, that’s exactly the kind of thing the Markey amendment would preclude.

The problem with the Markey amendment, specifically 715(b)(3), is that it says networks must provide equal access to improved quality of service (QoS) without imposition of a charge. Saying that someone must provide a product or service that has value without charging for it is unreasonable restraint of trade. If network companies are prevented from providing enhanced QoS in software, by tagging individual packets, they can only do so by building more physically separate leased lines. That would mean an even more Balkanized net with the Markey amendment than without it. That’s called the Law of Unintended Consequences biting zealots right where their groupthink comes from.

Let’s go back to the Delta/Northwest example for a moment. Another solution to the problem would be to tighten up the requirements for when the “peering arrangement” kicks in. Similarly, with regards to net neutrality, the solution is not to outlaw any kind of QoS without regard for its necessity or value to customers. Instead, the solution is to specify in greater detail how and when it may be applied. If we could define – in general terms – what criteria may or may not be used to assign QoS, what obligations network providers have regarding QoS values specified by others, what disclosures those providers must make regarding what bandwidth is actually available and how it has been assigned to different QoS categories, then we’d be onto something. Then network providers could write specifics into contracts with their customers and with other networks, without sacrificing the ability to offer new products and services or to price them according to market demand. That, unlike the Markey amendment, is a solution that will actually sustain growth and development of a free internet.

Oh No, Not Again

It looks like Auckland had another power outage. What is it about Auckland that prevents it from having reliable power? I particularly liked this quote.

Transpower’s Chris Roberts said 1000 megawatts of supply had been lost at a time on a winter’s morning when close to 2000 megawatts would normally be used.

Transpower presumed the power failure was weather related but could not rule out a maintenance-related problem, he said.

He said the incident was a rare one which occurred at the worst possible spot.

“These sort of incidents are probably going to occur once every 50 years, so its a matter of how far do you go.

OK, Mr. Roberts, so how do you explain two failures within 10 years? Equipment is going to fail. We all know that. That’s why intelligent people design such systems so that a single transient/localized failure doesn’t bring the whole system down. Oops, I guess I answered my own question. Since they failed to do that – failed twice – it’s clear that the people running Vector/Transpower/Mercury are not in fact intelligent enough to be in such important positions of public trust. Maybe they should be replaced by competent people this time.

Net Neutrality

The biggest topic throughout much of the blogosphere today has been so-called net neutrality, which is basically about whether or not bandwidth providers can charge different rates for different kinds of traffic. This issue can be “framed” in two very different ways, depending on which side you’re already on.

  • Without net neutrality, bandwidth providers will be the ones deciding which sites everyone can view, and/or charge extortionate “tolls” for net use. Innovation will suffer, startups will die, and we’ll all be cast back into the dark ages before you could buy your neighbors’ cast-off junk on eBay. This view is espoused mostly by content providers (such as eBay) and content-provider wannabes (e.g. bloggers trying to make money from AdWords).
  • With net neutrality, a surge of traffic to some entertainment site could damage the internet’s ability to deliver services essential to serious business or even life itself if emergency services’ voice-over-IP lines stopped working. Bandwidth terms and prices will be determined by Big Bad Government instead of supply demand, representing another setback for free markets yadda yadda yadda. This view is espoused mostly by the bandwidth providers and people who understand how the net actually works under the covers.

I’ve written about this from a political/economic perspective on It Affects You, but here I’d like to address a more technical issue – the attempt to represent net neutrality as a necessary extension of the end-to-end principle (E2E). This largely seems to stem from a comment by Lessig and McChesney in their article on “net-neut” (as it’s being called):

Net neutrality means simply that all like Internet content must be treated alike and move at the same speed over the network. The owners of the Internet’s wires cannot discriminate. This is the simple but brilliant “end-to-end” design of the Internet that has made it such a powerful force for economic and social good: All of the intelligence and control is held by producers and users, not the networks that connect them.

E2E does NOT require that all traffic be treated alike and moved at the same speed; the equivalence that they’re trying to establish is totally bogus. E2E is about the endpoints taking responsibility for their own needs (especially reliability) themselves as much as possible, but taking responsibility does not mean prohibiting others from doing what they need to or what they believe might help. It just means don’t make assumptions. Don’t assume that the underlying network will handle timeouts and retransmission, fragmentation and reassembly, checksums and everything else. If they do handle all of those things, great. You’ll have to do less work that way, but you had better be prepared to do more work if necessary.

As it turns out, networks can’t be – and already aren’t – based on a such a “totalitarian” form of E2E anyway. No matter how much responsibility endpoints take, for example, they can’t make up bandwidth that’s not there. They don’t even know how to route stuff. If the internet were based on totalitarian E2E, every node on it would have to specify every element of every packet’s path to a destination – what we network folks call source routing. Every node would need a routing table the size of Kansas, and the overhead of keeping every one up to date (and/or dealing with misrouted packets) would be more than the network could bear. That’s obviously not how things work. In actual fact, even the nodes that do have huge routing tables have to take “short cuts” such as BGP and shortest-prefix routing and MPLS and a very large bag of other tricks all of which involve the network being far more aware of the traffic passing through it than “treat alike and move at the same speed” would allow. That’s just not a design that can work. It’s taking a fundamentally good principle and perverting it by (mis)applying it in ways it wasn’t meant to be applied.

Reasonable people might or might not believe that net neutrality is necessary for the continued growth of the internet (not that growth is in and of itself necessarily a good thing). I happen to be among the dissenters, but I can accept that I’m in the minority. What I will not accept, though, is net-neutrality proponents trying to ride on E2E’s coat-tails when net neutrality and E2E really have little to do with one another. Why not just portray net neutrality as a way of protecting the poor little children while you’re at it? Better yet, find an argument that doesn’t rely so much on misrepresentation.

Lucky Girl

Most people in the US have probably never heard of Sophie Delezio, but maybe they should because her story is both a warning and an inspiration. The warning part is that she has been hit by cars twice, resulting in massive burns, the amputation of both feet, and a medically induced coma so she could heal other injuries. Remembering how anti-drunk-driving campaigns just seemed to be getting started in Australia four years ago, it makes me wonder about the state of awareness of child and auto safety over there. The inspiration part is that she and her family seem to have set an amazing example of courage in the face of such tremendous adversity. Look at the picture in the linked story. Notice all the smiles. If I had been involved in something like that, and knew (as the story mentions) that I still had plenty of surgery etc. to look forward to, I don’t think I could present such a positive image. My favorite part, though, is the way Dad manages to make something else positive out of the situation at the end.

But there was one more favour he wanted to ask: “When you do see people like Sophie out there, with any disability or any differences, please give them a smile, say hello to them, make them feel like they are citizens of our country.”

Now that’s class. Well done, sir.


In my last technical article, about adaptive readahead, I touched on a subject I’ve been meaning to write more about. Here’s the relevant passage.

Having multiple parts of a system each trying to be smart like this, when being smart means consuming resources while remaining oblivious to the others, often hurts overall performance and robustness as well.

I see this kind of “interference” between components quite a bit. In fact, I’d say it’s one of the most common marks of bad design. For example, as readers of my article on server design surely know, multiple components each trying to manage their own thread or memory pools for their own maximal benefit, without any kind of coordination, can often be a bad thing. As system-wide resources are overcommitted, all sorts of thrashing and starvation and other problems set in. When every component tries too hard to maximize its own performance, the result can often be lower performance throughout the system. I’ll resist the temptation to make a political point here.

Such interference is quite apparent in the data-storage world, in a couple of different ways. For example, every time an operating system changes the way it does readahead, it affects the access pattern seen by a disk array that might also be trying to do readahead. In some cases, this can lead to wrong decisions being made about which blocks to prefetch, wasting cache space and bandwidth that could otherwise be put to better use. Similarly, an operating system’s buffer or page cache does a pretty good job of turning a mostly-read access pattern into a mostly-write access pattern. If disk arrays’ caches were designed to optimize a mostly-read access pattern at the expense of a mostly-write pattern, this would also represent a kind of interference. It doesn’t happen because disk-array designers are not idiots, and it’s a little known or appreciated fact that those big caches on high-end disk arrays are mostly there to absorb writes. Caching is also involved in what is probably the most severe kind of interference as well – that which affects not just performance but correctness. In my experience, a frightening percentage of bugs are caused by programmers caching or copying data without having a strategy for how changes to separate copies will be reconciled. This can result either in stale data being returned to a user, or to system failure as different parts of the system make decisions based on what’s supposed to be the same information but isn’t. Caching is one of the most common techniques people use to improve performance, but the interference that often results can be a killer.

If interference is so bad, why is it so common? Usually it’s because one component is trying to do another component’s job, and the reasons for that in turn mostly come down to laziness and arrogance. Laziness is a factor because coordination requires work. It’s easier just to have your piece of code go off and do its own thing (e.g. with its own cache) without trying to coordinate with anything else. Even better, if the Frobnicator component is currently the bottleneck in the system then the guy who adds a standalone cache to it can get credit for fixing the bottleneck despite the fact that they might have hurt overall performance or even introduced serious bugs. “The Frobnicator is no longer the bottleneck. Great job, Joe! Here’s a bonus.” Yeah, thanks Joe, for making everyone else’s job harder. Attempts to save a few cycles out of millions by making zillions of direct function calls instead of the few dispatch tables and callbacks of a well-defined API fall into the same category. The arrogance part of this is our old friend the “Not Invented Here” syndrome. The most common rationalization for embedding redundant functionality into a component is “this way we can get exactly the XXX functionality we need without all the overhead of stuff that’s there for someone else” where XXX is the component that should be external. Sometimes this also comes down to laziness; just as it’s often easier to rewrite code (and reintroduce already-fixed bugs) than to fix it, it’s often easier to reimplement another component instead of extending it. Other times it’s just because the filesystem guys think they can write a better volume manager than the volume-manager guys, or the database guys think they can write a better block cache than the operating-system guys. They’re usually wrong, but programmers – especially those who have developed a high level of skill without self-discipline to match – love trying to outdo one another. Unfortunately, it’s their fellow programmers and users who suffer the consequences.

The thing that kills me about all this is that decomposing a complex system into the right set of components with the right interfaces between them in a way that minimizes interference (or potential for interference) is the very essence of an architect or designer’s job. If you can’t figure out a way to make generally useful functionality generally available, then you’re not doing a very good job as an architect or designer and should have any such title removed from your business card. If a significant part of your code is copied (either literally or conceptually) from somewhere else, or if you often find yourself fixing what’s basically the same bug in multiple places, or if every code change seems to require re-tuning everything to fix performance, you’re probably suffering the effects of interference. If you’re smart, you’ll figure out a way to refactor so that the system “flows” more coherently instead of fighting against itself.

Amy is Two

Saturday was Amy’s second birthday. It was a pretty low-key affair, basically just a few family to “help” Amy open her presents, followed by carrot cake and banana ice cream. I don’t really have anything to say that’s any different than last year’s tribute, so I thought it would be fun to compare some pictures. Here’s the main picture collection from when Amy was born, and here’s her first birthday. Lastly, here are a couple of pictures from Saturday.

Amy and Cindy Taking advantage of the “flash effect”; often I can get the best expressions if I take a picture right after someone else did. It’s amazing to me how mature Amy can look sometimes.
Amy with toys Amy with some of her new toys. She now has her very own watering can so she doesn’t keep walking off with Mommy’s, and the caterpillar (“bi(g) bi(g) pilla”) might help with counting.

I guess it’s worth saying a few more words about counting. Amy knows all of the numerals (including zero), she can distinguish between having one or two or sometimes three objects in view or in her hands, and she seems to understand the idea of counting as an activity. What she doesn’t quite get is the idea that when you’re counting you’re supposed to do the numbers in a particular sequence. Instead, she just kind of picks and repeats her favorites. “Two, five, six, five, six, five, six!” is a common example, with the last “six” said with obvious pride. Lately I’ve been hearing “Eight, nine, eight, nine!” as well. I like to think that she’s just rejecting confinement to the standard sequence that the rest of us with less imagination use.

I have several more pictures that I haven’t processed yet, and some video too. Stay tuned.