Guarding the Home Front

One part of my job has to do with sales. Historically this has been mostly post-sales, but I’m increasingly – and welcomely – being consulted about storage configurations during the pre-sales phase as well. I often find myself having to be quite aggressive in rejecting underprovisioned or unqualified solutions, because of a very simple fact:

Customers are often ignorant or unreasonable, and it’s my job to be a counterbalance.

When I say that customers are ignorant or unreasonable, I mean no insult. Storage is probably the least commonly understood part of most systems. The people who buy systems tend to understand hardware and application software fairly well, networking a little, and storage least of all. A parallel filesystem, which combines storage and networking and parallel/distributed programming, is something most people are simply unable to conceptualize properly. Customers have their own problems to solve. If they don’t understand ours, even when ours affect how they solve theirs, they simply need to be educated.

Customers being unreasonable, though, is a different problem. The customer we see is often not the user, but rather some IT/procurement professional. Nonetheless, their primary duty is to their users and they serve that duty by demanding maximum performance for minimum cost. They might know that storage often accounts for 30-60% of total system cost in commercial environments, or 10-20% in HPC, but they’ll still try to bargain that down aggressively. Initial demands for storage performance commensurate with compute performance for only 5% of total system cost are not uncommon, and I’ve even seen 1%. So, what happens if I come back with a configuration that’s more like 15%? The salespeople, who are in the middle of all this, then get stuck in the technical equivalent of the Overton Window in politics. They see one side offering an extreme position and the other offering a moderate one, leaving them with a range of choices that’s all on one side of the true median. If they try to compromise by splitting the difference, they’re likely to end up with a solution that increases support costs and leaves customers with a bad impression.

So, what’s the solution? Well, if the customer is going to be an aggressive advocate for maximum performance and minimum cost, then I must be the exact opposite. My proposal should therefore not be a 50th-percentile best guess, but a 99th-percentile solution involving the most hardware in only the most thoroughly tested configurations. It’s like our system of justice, in which the prosecution and defense are not only expected but required to offer the strongest possible advocacy for opposing positions so long as they adhere to the rules and the facts. Only by being an absolute “Dr. No” when it comes to evaluating proposed configurations can I, or anyone in an equivalent position, expect the end result to be a configuration that both satisfies the customer and avoids costly support interventions later.

Clues for the Clueless

Most people will have no idea what I’m talking about. That’s OK, because only two really need to recognize what I’m talking about. Just so nobody misunderstands, neither of those people are or ever were at SiCortex, and neither ever will be if I have anything to say about it. Their crimes were committed elsewhere.

  • If it’s your component that needed to be rewritten three times, and you’re the one who quit minutes ahead of being fired for actively working against company interests, you don’t get to complain about “a product that never worked.” Also, the product did work despite your efforts to sabotage it.
  • If you have made a career out of riding others’ coat-tails, running multiple startups into the ground, and generally presiding over more layoffs than hires, you don’t get to brag about how you’ve worked so hard all your life creating value. Also, threatening to move to Europe because the US has become too socialist just makes you look stupid.

It’s too bad people like you ruined a perfectly good company. You should be the ones on the unemployment lines. No, scratch that. You shouldn’t be collecting unemployment. You should be paying anyone whose unemployment is to any degree attributable to your considerable shortcomings.

Corporate Tax Basics

On the way in to work today, I heard some Republican gasbag repeat two common misrepresentations about corporate taxes.

“The US has the highest marginal corporate-tax rates in the world.
So what? It’s not the marginal rates that matter; it’s the effective rates. A high marginal rate with a high threshold and many exemptions will result in less tax (and thus revenue) than a lower rate might. In fact, the Center for Budget and Policy Priorities has studied this matter, and they report that the US effective rate of 13.4% is lower than the OECD average of 16.1%. Somebody on the corporatist side of the fence had better come up with some credible new numbers before they repeat such misleading statements.
“Corporations don’t pay tax, they pass it on to consumers.”
They might pass some on to consumers, but not all. In a competitive market, raising prices might not be an option, so instead the cost gets passed on to employees, suppliers, etc. It might also result in corners being cut in product safety, environmental impact, and so on. Given that all of these other affected parties might in fact be overseas, the idea that corporate tax in the US gets passed straight on to consumers in the US becomes pretty questionable. Of course, both the original claim and my counterclaim are just theories. Where are the numbers showing the actual observable effect of past changes in corporate tax rates on prices (and wages or employment which we also shouldn’t ignore)? I can’t find any easily, suggesting that no clear measurable effect has been documented – i.e. that the compensating mechanisms I mention negate practically all of the assumed effect on consumer prices. Those making the “passed on” claim should prove it, and they never even seem to try.

This whole idea that we should coddle corporations because markets are good is based on two basic errors. The first is conflating corporations with markets. Sorry, but people are part of the market too – people working, people consuming, people running sole proprietorships and partnerships and other businesses where they still pay personal rather than corporate tax. Taxing corporations differently than people is simply giving one class of market participant an advantage over another. One could argue that no such distinction should be made, or that the difference should go the other way – i.e. that capital as represented by corporations should be taxed more than labor as represented by flesh-and-blood people – but I’ve yet to see a reasonable argument for taxing corporations less. The other mistake is a little harder to explain, but even more important. One of the most excellent books I’ve read recently is David Bollier’s Silent Theft, and one of the most powerful statements in it is this (from memory):

The market should be inscribed within the commons, not the other way around.

What he’s saying is that resources – he talks elsewhere in the book mostly about things like land and other natural resources, or “intellectual property” – should be assumed to be shared unless they are explicitly privatized, monetized, etc. What we tend to do is the exact opposite; everything is assumed to be private unless explicitly placed in the public domain. As Bollier explains far better than I could, that model doesn’t really work even a majority of the time. Markets are powerful tools, and perhaps a majority of property should be handled via markets, but that majority should be explicit and not just the result of a mistaken “private is good” assumption.

Bollier’s sentiment is very closely related to another that I’ve heard expressed, and with which I completely agree: markets should serve people, not the other way around. Somehow we’ve become a society where people feel that their duty is to be good workers and good consumers so that “the market” may thrive. That’s garbage. Again, markets are powerful tools, but they are only a means to an end and that end is human prosperity. When markets fail (we should all know by now that they do for a variety of predictable reasons), or when they “succeed” in a way that has a net negative effect on the life experiences of real flesh-and-blood people, then they are not the right means to the right end and should be eschewed in favor of more appropriate tools. Sometimes that tool is government, sometimes it’s individuals acting in a context that is neither governmental nor corporate, sometimes it’s even nature (or perhaps even God for those inclined that way). In any case, these other actors must sometimes be allowed to take a primary role instead of being subordinated to a market that effectively (and too often legally) puts corporations first. Like governments, markets should be of the people, by the people, for the people. We have a long way to go before that’s the case.

Sunday Video 2009-02-22: Christmas 2008

I apologize for the fact that one is sideways and the other two are of poor quality. As for the sideways, I still forget sometimes that video editing programs unlike still-image editing programs – still treat a simple ninety-degree rotation as an impossible task. It’s not like anybody could figure out what pixels are actually displayed on the screen for each frame, rotate that ninety degrees, and re-do all of the compression the same way as was originally done when the pixels were coming in from an image sensor. Nahhh, that would require a few minutes’ work to help users. Unfortunately, I don’t have any similar excuse for the image quality in the other two. Normally this camera shoots pretty decent video, and the lighting conditions didn’t seem all that challenging, but the video quality is still so poor that I debated not even posting them.

opening the present Amy opening and then playing with her favorite present. (8.8MB)
clowning around Trying to get Amy, Oliver, and Eli to pose properly for the camera. Eli took “hat up” very literally. We never did really get the shot we wanted, at least not while the video was running. (1.8MB)
walking Frances was actually quite a cooperative subject, and early-walking video always seems worthwhile. (1.2MB)

Home IT Update

A few weeks ago, I wrote about my new QNAP TS-109. Since then, its use has only increased. I now regularly mount it from both my laptop and the upstairs desktop, and use the built-in BitTorrent client to download stuff unattended off the web. It’s really nice to have it be the only device on during an overnight download. Last night, I set it up as a print server, usable from both laptops and the desktop. This turned out to be pretty trivial on both Windows and Linux, and provides an office-like convenience we’ve never had in the house before. Another serendipitous use has been to plug my iPod Touch into it; since nothing else in the office is on any more, that’s the only live USB port most of the time.

One thing I had mentioned in my previous review is that the TS-109 really highlighted the shortcomings of the 100Mb/s (MTU=1500) switch inside my old wireless access point. I actually got the new one before my Boulder trip, but tonight was the first chance I had to play with it. It’s a Netgear WNR854T, which has MIMO, draft 802.11n, and four gigabit Ethernet ports, for less than $50 including shipping from eBay. Not everyone likes Netgear, but my previous access point was a WGR614 and it worked fine with better range than the D-Link it replaced or the Linksys that came with my Verizon service. Like moving this website from one host to another, upgrading from one access point to another is something I’ve gotten pretty good at. Here are the steps.

  1. I saved the configuration from the old AP, and also manually wrote down every bit I could find that I thought could possibly matter. For example, the MAC address from the old AP turned out to be handy.
  2. I plugged the new AP in, and connected my desktop to it.
  3. I started setting up a parallel configuration on the new AP. Where the old SSID was xx2, the new one is xx3. Where the old IP address range was xx.xx.2.xx, the new one is xx.xx.3.xx.
  4. Plugged the TS-109 into the new AP, tested functionality between that and the desktop.
  5. Modifed my laptop’s configuration to use the new AP, tested functionality between that and the TS-109.
  6. Plugged the new AP into the old one, creating a double-NAT daisy chain, and verified that my machines could connect to the internet.
  7. Modified Cindy’s laptop configuration to use the new AP, and tested that.
  8. Took the old AP out of the loop, plugging the outside connection directly into the new AP.

Up to this point, everything had pretty much just worked, and all the repeated testing was just me being careful. It turns out that the new AP couldn’t get an IP address from Verizon until I cloned the old AP’s MAC address. This is probably just a measure to prevent people from abusing their connections by connecting lots of devices simultaneously, and might well have resolved itself after a timeout, but I didn’t feel like waiting. With this one change, just about everything worked. The only other issue is that everything was still using the “cloned” IP address range xx.x.3.xx, which would have required changing network mounts and printer configurations, so I changed the IP address range on the AP to be the same xx.xx.2.xx that the old one had used. I was pleasantly surprised to find that all of the address-reservation and port-forwarding entries were automatically update to reflect the new subnet, so I didn’t have to go through all that again. There was nothing left to do now but decommission the old AP, so now it’s sitting in a drawer with its predecessors.

So, what’s the verdict? Mostly I don’t notice any difference, and that’s good. The one negative difference is that the new AP seems to require reboots for a lot more configuration changes than the old one did. That’s both surprising, considering that it’s a newer product from the same vendor, and annoying. On the positive side, things do seem a bit snappier but the only quantitative measurement I’ve done is iozone between my laptop and the TS-109 via NFS. 2.4MB/s (19Mb/s) is certainly better than it had been before, but really only reflects on the wireless speed and signal quality. Some day I’ll have to run a similar test over the wired connection from the desktop, since that was really the case that I had in mind when I decided to upgrade, but I had been in Windows (for iTunes) and didn’t feel like rebooting into Linux just for that test. I guess the net result is that I don’t seem to have made things worse and there’s good reason to believe that some usage scenarios – especially video conversion on the desktop with both source and destination on the TS-109 – will be a lot better. That’s pretty good for $44 and a couple of hours.

Best/Worst Band Names

Last night at dinner, apropos of approximately nothing, I came up with the idea of “Show Canceled” as the possible worst band name. The fliers would not help bring in much of an audience, I think. Then we got to wondering about the best band names based on such considerations. I came up with “No Cover” first, but I think “Free Beer” would be even better. A band with that name could tour college campuses and be assured to sell out every venue, I’m sure.

P.S. Now I remember how it came up. We were talking about WALL-E, which has been Amy’s favorite movie for a few weeks. The phrase “Foreign Contaminant” features prominently, and also struck me as an amusing band name.

Flash of Genius

This is the current movie on United’s westbound domestic flights, and so I watched it on the way to Boulder. It was rather painful for me – as both an engineer/innovator and a husband/father – to watch. In some ways it just hits too close to home. Mostly that means it’s a good movie, depicting realistic people and situations, and I did enjoy it. I do have one pretty major quibble with it, though. The movie clearly tries to depict Ford as having deliberately copied protagonist Bob Kearns’s idea from a prototype, but the actual jury decision was non-deliberate infringement of Kearns’s patents. This raises the issue of how Kearns, who had turned down multiple million-dollar offers when a million dollars actually meant something because he wanted an admission of wrongdoing from Ford, would consider such a decision adequate. Instead of dealing with that, though, the movie just ignores the disconnect between what they portray Kearns as wanting and what he actually got.

There are also some areas in which I found myself losing sympathy for Kearns. For example, there’s one point where he’s being offered a settlement. His wife is clearly in favor, but he won’t even discuss it. In my opinion, that’s no way to run a marriage. When Cindy and I wrote our own wedding vows, one was to treat each other’s needs and priorities as equal to our own. That’s a high standard, which neither of us has always met, but I still think it’s a good principle and one at least as important as the principles on which Kearns’s refusal of the settlement was based. I think his behavior (at least as portrayed in the movie) was contemptuous of his wife and his marriage, and it’s no wonder she left him.

The other point on which my views diverge from Kearns’s is on his insistence that Ford admit wrongdoing. As one of the other characters points out – rightly in my opinion, despite that character being more of a bad guy – the way companies admit wrongdoing is by paying cash. Car companies in particular do this even when lives are at stake, when cars burst into flame or roll over and crush people. Other companies also pay for what they do wrong without ever admitting it was wrong, as when Sun refuses to admit that they drew inspiration for ZFS from NetApp’s WAFL (even as they aggressively take the exact opposite position when they demand that anyone who does anything in the system-tracing space must bow down and worship DTrace first). Companies measure their value in dollars, they both collect and pay their debts in dollars, and expecting them to provide any kind of non-monetary compensation for harms they have caused is just not realistic. In fact Ford never did admit wrongdoing in the Kearns case. Kearns did eventually prevail, but the court’s award was less than had been offered before. More importantly, how many other inventors in similar situations didn’t prevail? We hear about the few that do, but in general I don’t think it’s a good example to follow. Make them pay what they owe, certainly, but don’t demand payment in a coin that they don’t understand.

Bird Fight

I look at this picture (via Pharyngula, who got it from National Geographic, and I wonder: how did we get to this position?

I mean, fighting in mid-air is unusual enough, but then what was the red-headed doing that enabled the flicker to grab its tongue?

Hill Climbing, Traffic, and the 2-4-6 Game

Imagine for a moment that you’ve been dropped into some random spot in the Himalayas. Your goal is to reach the top of Everest. If you just keep going up at every opportunity, will you reach it? Probably not. Ignoring for a moment all the issues of food, oxygen, and human strength, there’s another problem: you probably landed closer to some other mountain. If you insist on going up, you’ll probably ascend that mountain, with Everest still towering above you and no way to reach it without going down first. You will have reached what was called a “local maximum” in your freshman math class. Since any optimization problem can be viewed as finding a path through the graph defined by same function, this same hill climbing behavior and this same potential to reach a dead end occurs in many other fields from the technical to the political.

Another concept similar to that of a local maximum is from game theory – a “Nash equilibrium” is a set of strategies for all players where no player can improve their own outcome by changing only their own strategy. To show how broadly applicable such concepts are, consider the article which brought all of this to mind for me – Removing Roads and Traffic Lights Speeds Urban Travel.

Using hypothetical and real-world road networks, they explain that drivers seeking the shortest route to a given destination eventually reach what is known as the Nash equilibrium, in which no single driver can do any better by changing his or her strategy unilaterally. The problem is that the Nash equilibrium is less efficient than the equilibrium reached when drivers act unselfishly—that is, when they coordinate their movements to benefit the entire group.

The solution hinges on Braess’s paradox, Gastner says. “Because selfish drivers optimize a wrong function, they can be led to a better solution if you remove some of the network links,” he explains. Why? In part because closing roads makes it more difficult for individual drivers to choose the best (and most selfish) route. In the Boston example, Gastner’s team found that six possible road closures, including parts of Charles and Main streets, would reduce the delay under the selfish-driving scenario. (The street closures would not slow drivers if they were behaving unselfishly.)

The game-theory version of this problem is a bit more like a maze than a mountain climb, with certain directions precluded by the self-interest criterion rather than by a limited field of view, but the potential for a hill-climbing approach to lead to a dead end is essentially the same. Anybody who actually paid attention in that freshman math class would know that more complex functions tend to have more local maxima (and minima). Since there are few functions as complex as those describing economic behavior, the graph of any economic function tends to be especially fraught with dead ends. This is why so much of economics is BS. A statement that “going in this direction leads upward” might be true in a purely local context cherry-picked by the person making the statement, but that truth might also be utterly useless in the part of the graph that actually reflects reality. In particular, understanding these freshman-level mathematical concepts blows a huge hole in the claim that individuals pursuing self-interest will result in a globally optimal outcome such as widespread economic prosperity.

To illustrate this divergence between selfishness and optimality, I invented a little game called the “2-4-6″ game. Two players roll dice a fixed number of times, adding each roll to their score, but with two twists.

  1. If the last digit of a player’s current score is 2, 4, or 6, they must roll exactly that number to increase their score. Any other roll leaves them stuck.
  2. A player who is not stuck (either before or after rolling) may “donate” one point from their roll to someone who is stuck, to get them unstuck. Such donation cannot result in the donor becoming stuck, even if the result is a score ending in 2, 4, or 6.

The important dynamic here is that donating always involves both an absolute and relative loss for the donor, compared to not donating. The donor loses one point, and the recipient gains at least two – from an effective roll of zero to their original roll plus one. However, this also means that the total score for both players will always increase as well. I’ve even written a Python script to simulate the results, both when players choose to help each other and when they decline (as the “selfish is good” contingent would claim is best). The result is that players in the selfish variant usually get a score of about 125 for 100 rolls, whereas players in the altruistic variant usually get about 140. The most important difference is this: even the “losers” in the altruistic variant do better than the “winners” in the selfish one.

The moral, of course, is that it’s always important to consider not only the local effects of a decision, but also what kind of system a multitude of similar decisions will create. Whether the subject is throughput in a congested computer network or GDP in a national economy, a system in which individual actors sometimes “sacrifice” immediate self-interest for the sake of creating or maintaining a better system often yields better results for everyone including those making the so-called sacrifice.

It’s A Wonderful Life

So, I’m going to Boulder again. As Vincent Rubio would say: don’t ask, don’t ask. I’m literally sitting at Logan Airport, waiting for my flight through Denver, and what do you think I just heard on the TV? That president Obama is going to be signing the Economic Recovery and Reinvestment Act – great acronym, BTW – in Denver tomorrow. I’m sure that will have nothing but good effects on traffic and travel throughout the region while I’m there.