Coding Survival Tips

A while back, I wrote a long article about server design. To this day, it seems to get a steady stream of hits, often showing up in log summaries right behind my platypus pictures and the Lord of the Rings personality test (don’t ask). Now I’d like to dive a little deeper and talk about actual coding. Here are some survival tips that I’ve picked up here and there. They’re not especially innovative or anything, but you’d be amazed how many otherwise-smart people cause themselves unnecessary pain by forgetting them. There’s also a little bit of an emphasis on writing code that doesn’t just work once, but keeps working for the long term, even after some inferior programmer has messed with it and then it’s out in the field where you’re forced to debug without the full suite of programming tools available.

Log Properly

One of the most critical debugging tools for code outside of the lab is a decent logging facility. Three particular properties are important:

  • It must have a minimal impact on performance, so it doesn’t hide timing-related problems.
  • It must be shared across all modules in the system, or at least on one node if your system happens to be distributed. This implies both thread safety and consistent timestamps; events must appear in the log in exactly the same order that they actually occurred.
  • It must allow run-time filtering of what gets logged. Debug on/off is not sufficient, and neither are debug levels. To avoid the twin evils of insufficient information and information overload, the person looking at a problem must be able to fine-tune the logging to at least the module level. This can be achieved by maintaining a simple bitmask of what’s being logged, one bit per module or logging point within a module.

Some systems provide pretty good logging facilities already; believe it or not, AIX is particularly good in this regard. Use what’s there if it has the necessary features. If not, it’s pretty easy to write one yourself – even for kernel use. Circular buffers are good, but make sure they’re large enough to capture a decent amount of activity.

Don’t Abuse the Stack

The most obvious reason for not abusing the stack is because your code might run in stack-limited environments, such as an OS kernel or embedded system. However, I do realize that some of the people reading this might be stuck writing wimpy user-level general-purpose-OS code (that’s a joke, son) so that’s not the real reason I advise against keeping lots of state on the stack. The real reason is that keeping too much state on the stack is just harder to debug. Crawling the stack can be error-prone even for a debugger (if you happen to have one handy when that nasty bug hits), and you can still get stuck trying to piece together what happened from state splattered across half a dozen stack frames. Yech. Most state belongs in standalone objects or structures. If you’re handling some kind of request, put all state for that request in a neat little bundle, not in arguments everywhere. Every module or function that’s involved in processing the request is then guaranteed to know everything they need to know, and you won’t have to go back time after time to add a new parameter through three levels of function calls so the one at the end can make a correct decision. This approach also makes it easier to look at dumps, either the raw-memory kind or the formatted dump-routine kind, and sooner or later that’s all you’ll have to look at while a customer – or your boss – is breathing down your neck.

Explicit state can be overdone. It’s often associated with a state-machine approach, which has a great deal to recommend it, but it can be a real pain to write loops without using local variables as indices (for example). There’s a definite judgement call involved, but it’s usually best to err on the side of putting/passing state in structures instead of using arguments and local variables everywhere. It’s usually more efficient, too.

Exercisers and Shims

We all know we’re supposed to keep modules separate, right? Right? Good. What’s less universally understood is that writing the code for the module itself is only half the job of producing a quality module that you or your other team members can use with confidence. If you think of a module as having an “upper” interface through which others call it, plus one or more “lower” interfaces through which it calls others, the other half of the job is writing an exerciser for the top half and shims for the bottom half to simulate other modules. I usually write the exerciser as a Python extension module, with Python objects corresponding to objects within the module, so I have the full benefit of Python’s data and control structures when I’m writing unit-test scripts or poking at things manually. Yes, that applies even to code that’s supposed to run in the kernel; it ultimately takes less time to write the exerciser than would otherwise be spent in that compile-debug-reboot cycle familiar to us all. In this particular case the shims have to include somewhat-accurate equivalents for the kernel functions that don’t exist at user level, and that can be tedious, but there’s no such thing as a free lunch. It helps you appreciate and understand your dependencies, at any rate, and that’s often helpful.

If you structure everything right, so that all the proper exercisers and shims exist for your modules, you should be able to test any contiguous subset of modules in your system without having to run the rest. This makes putting everything together – and actually having it work – a breeze compared to the common approach of testing everything by itself and then throwing the whole lot in one big blender. There’s also nothing wrong with having exercisers talk to shims, e.g. for error injection. The net result is a faster debug cycle, more complete test coverage, and less pain when other people find bugs in your code.


Today’s topic is logorrhea, which is variously defined as “excessive use of words” or “pathologically excessive talking”. Its most common manifestation is as a belief that more words make any argument stronger. This particular strain of logorrhea often gets cured somewhere in freshman or sophomore year at any decent university, but sometimes it persists well into adulthood. As many of you have probably predicted, the examples for our discussion will be drawn from that logorrhea poster child Steven Den Beste, and particularly today’s 2200-word screed.

Steven wrote what he apparently thought was the last word on strategy for the war on terror, which weighed in at over 6000 words in outline form. He was very proud of it, and perhaps he should be, but it shouldn’t be too much of a surprise that some people disagreed with both his assumptions and his conclusions. Steven even provides a fairly lengthy excerpt from Patrick Schaefer poking holes in various parts of his arguments. He then provides a stunningly dishonest paraphrase of Patrick’s commentary (odd for one who objected so strongly when D-squared provided far more accurate paraphrases of his own posts)…and then he gets mad. Here’s part of his response:

let’s try a little thought experiment. Let’s schedule a debate, and invite a lot of voters. The first speaker stands up and makes a case for one position, laying out his explanation of why the problem happened, and then saying what he thinks needs to be done to solve it, and explaining why he thinks it will help. Then he sits down.

His opponent, on the left side of the stage, stands up, grins at the audience, and pulls his pants down and moons the first speaker. He then returns his pants to their customary position and returns to his seat. End of debate.

If the audience was not partisan ahead of time, which advocate is more likely to have convinced them?

I’d call that a pretty bad analogy. Does Patrick’s response strike anyone as mere mooning? You might disagree with it (I do, in fact) but it does seem fairly articulate and well-considered. Sometimes, when someone just stands up and babbles inanely for half an hour, I’d say they deserve nothing better than mooning, but many of Steven’s interlocutors (not all; he did manage to cherry-pick a few examples of true mooning) did far more than that. Nonetheless, Steven dismisses every one of them as though they were nothing but pants-droppers. He derides anyone to the left of himself and Attila the Hun as part of the “the post-modernist lit-crit left” even as he denies his own association with the xenophobic paleolithic right. Most notably, he never actually tries to address the objections raised by Patrick and others.

The point here is not that Steven is a hypocrite, even though he is. The point is that the sheer length of Steven’s non-answer works against him. The man just can’t pass up a chance to insert (what he thinks is) a pithy comment, even if the five paragraphs of setup turn a would-be zinger into a predictable groaner. In the hurry to insert the pet phrase, he turns a blind eye to both reason and irony. What other explanation could there be for his accusing others of “preaching to the choir” or “refusing to consider the idea that they might need to engage in cogent debate”? His main point, about people who eschew honest debate in favor of personal attacks, could have been made in two or three paragraphs without falling into a dozen potholes of his own devising.

I challenge anyone to read the post in question, and not conclude that its length hurts rather than helps the author’s cause. Whining about personal attacks while indulging in a few of one’s own does not improve with repetition.

Where’s George?

Now that we’re being told the war in Iraq is grounded in concern for the oppressed people in Iraq and not weapons of mass destruction, I have to wonder: when is George going to visit the people for whom he feels so deeply…or our own soldiers? After all, major hostilities ceased nearly three months ago, right? “We’ve got the force necessary to deal with the security situation” in his own words. In our egalitarian society, our leader is supposed to be one of us, not a separate being insulated from the concerns and fears the rest of us face. Why is “bring it on” OK when it’s mere soldiers’ lives on the line, but Bush himself won’t even land on the deck of an aircraft carrier until it’s half a world away from Iraq?

Why Rules Matter

One of the conversations on America’s Debate recently was on the topic of whether certain kinds of taxation can be considered theft. It’s a common enough meme. Many people seem to feel that government is like a store, where every individual taxpayer is entitled to at least a dollar’s worth of goods or services for every dollar paid in taxes. Nobody complains if they get more, of course, so it all ends up a little like Lake Wobegon (“where every child is above average”), but there’s no shortage of people who whine about getting less. Clue #1: government is not like a store. The government is responsible to the people collectively, not individually, and sometimes benefits do not flow back to the same people who paid the taxes to create them. I used the example of fire or ambulance service. It doesn’t matter that the direct benefits are not distributed evenly, or that many of the people who do receive direct benefits are destitute; we’re all better off for having such services available. Likewise we’re better off having laws and courts that enforce contracts, protect us from thieves, and generally provide a business environment in which markets are truly free and competition can thrive. I might not benefit directly from apprehension of a particular thief, but I do benefit from the fact that thieves in general get caught.

This brings me to my real point, which is far more general. Regular readers know that I’m a big fan of game theory, and particularly the Prisoners’ Dilemma. One of the most interesting results in game theory, particularly in the work of Robert Axelrod (who has written two books and numerous articles on the subject), is the idea of cooperation between players arising spontaneously, i.e. without explicit collusion or even communication. What’s often overlooked is that spontaneous cooperation is only one example of a more general case: the success of a strategy based on heuristics rather than concrete knowledge about outcomes. You might not know, in any particular case, whether doing X will work out well, but you certainly can know that if you always do X you’ll do better over time than if you always avoid doing X. It’s a type of reasoning very familiar to game players. Without being able to foresee exactly how it will play out in the current game, a player might very well know that allowing a particular exchange or pawn structure to occur just seems to invite disaster time after time…so s/he avoids it. A particularly interesting example is the “poisoned pawn” that shows up in many openings. It’s hard to explain to a non-player, but the basic idea is that sometimes taking what seems to be a free pawn can be a grievous error. Sometimes the error is obvious, such as when a bishop or queen gets trapped. Other times the error is extremely subtle, such as when removing the pawn opens up a powerful line of attack for the opponent twenty moves later (b- and g-pawns are notorious this way).

The same sort of reasoning is just as applicable in human affairs. Certain behaviors are simply self-defeating, not because of immediate or direct retaliation but because they upset the social order in a way that just always seems to rebound on the original miscreant. There’s a reason every religion and culture has come up with ideas like karma and the Golden Rule and “as ye sow” and so on. They’re the real-life equivalent of the chess player’s “I get screwed whenever I do that” knowledge. There’s often a threshold effect too, best illustrated by a lesser known example from game theory – Rousseau’s Stag Hunt. A group of people want to hunt a stag, which can only be hunted successfully by a group acting in concert, and then only sometimes. But there are also rabbits in the woods which, while valued less than stags, can almost certainly be caught even by a single individual…especially if everybody else is off hunting stags. If everyone stays committed to the stag hunt, a feast is likely (though not certain). If too many people leave the group to hunt rabbits, though, there’s no chance of a venison feast. (Some of you might recognize this as being very similar to the more recent Tragedy of the Commons.) The point here is that societies and governments are more like stag hunts than anything else. Everyone thinks they’re the only one hunting rabbits, so they think it’s OK, but when everyone’s hunting rabbits we all get cheated out of a stag. The laissez-fairies are not the kings of competition, as they like to portray themselves; they’re the kings of degrading competition, so they can “get ahead” on a shorter, poorer track.

The systemic effects of either selfishness or selflessness far exceed anybody’s ability to predict consequences. Sometimes it’s worth adhering to a rule not because a reward is anticipated but because thousands of years of experience have taught us that it’s the right thing to do. We might not understand exactly why, it might seem less compelling than our rationalizations for selfishness, but it’s the right thing in the long run nevertheless.

Krispy Kreme

I tried my first Krispy Kreme donut this morning – still warm, even. Those things are yummy. I can definitely understand why people go so crazy over them, even though I can also see limits to their appeal. As yummy as they are, I can’t imagine wanting one every day.

Broken Code

One of my pet peeves lately has been the “not a big deal” excuse used by some developers to resist fixing things that make their code annoying to work with. This most often seems to appear in configuration/setup contexts, where it comes out as “it only takes a few minutes to configure that”. Well, first, no it doesn’t. It only takes a few minutes each time someone has to do it. If it costs your fellow developers five minutes, for example, times thirty developers times ten times a month every time they install something, that’s over four days’ worth of time lost every month. If you’re an open-source developer and your code is used by hundreds of thousands of users, that same five minutes each time can mount up into dozens of person-years. Linux installation and configuration is notorious for being unnecessarily time-consuming. The costs probably add up to more than the entire annual productivity of a rather large city just because some folks were too lazy to fix some glaring problems. And that’s all if things go right. If you make it easy to miss a step or do it wrong, each user might get to spend hours or days debugging “spooky” problems that result. The cost even for a very small user base can actually be quite large.

My other answer is that it doesn’t matter how little time it costs when that number should be zero. Would you say “it’s only a small turd on the kitchen table”? Of course not. Some things just shouldn’t happen, and whether they’re frequent or rare, big or small, matters less than whether they happen at all. Don’t leave turds in your code.

Bring the Boys Back Home

It looks like the Loyalists have suddenly discovered the feelings of the grunts doing the fighting in Iraq, and have started screaming for those grunts to be brought home. Undoubtedly the screaming will get ever more strident until the administration “bows to popular pressure” and brings the troops home, leaving an Iraq with a shattered civic and political infrastructure to collapse into chaos and ruin. All according to plan. They’ll probably even use the withdrawal as an excuse for never finding the WMD. “We had to get out of there, we never had a proper chance to search.” Yeah, right.

What galls me is that this is why they shouldn’t have gone in the first place. We knew that Iraq would be a mess after we’d pummeled them, a humanitarian and geopolitical disaster in the making. We knew that the Bush administration wouldn’t have the patience to see the job through. They don’t want the situation to stabilize, and they don’t want to normalize relations with the locals. They want to make sure their hand-picked interim government can win the civil war (heaven forbid we should try to prevent one), then get our troops out and ready for the next neocon misadventure. This new-found respect for the members of the military and their families seems just a tad cynical and self-serving to me. Why weren’t their lives and well-being important three months ago, hmmmm?

Wild Times

Since the weather has been nice, I’ve started going on little mini-breaks from work in the midafternoon. There’s a road near the office with a lot of undeveloped land beside it that’s gone to meadow and forest, so it’s good for watching and listening to birds. Today when I went out, I heard a song sparrow right away, then I watched two goldfinches chase each other around. I sat down in my favorite shady spot, and heard a woodpecker. A moment later I noticed that some blackberry bushes nearby seemed to have ripe berries, so I picked a handful and brought (most of) them back to share.

The cool thing is that this was all in the space less than ten minutes. Could there be a better way to take a break at the office?

Platypus Pickup

Rys McCusker gave me an idea for painting my car…or maybe my house, which actually does need a paint job. Thanks, Rys!

Anonymity Online

Clay Shirky has written an interesting essay called A Group Is Its Own Worst Enemy about “social software” and group behavior. There are lots of interesting points in there, and he does mention good old LambaMOO, but here’s my favorite part:

anonymity doesn’t work well in group settings, because “who said what when” is the minimum requirement for having a conversation. What’s less well understood is that weak pseudonymity doesn’t work well, either. Because I need to associate who’s saying something to me now with previous conversations….
…you have to design a way for there to be members in good standing. Have to design some way in which good works get recognized. The minimal way is, posts appear with identity. You can do more sophisticated things like having formal karma or “member since.”

Hear, hear. Weak pseudonymity has been the death of too many sites to mention, most notably (for me) e-thepeople. The near-total lack of consequences for behaving badly (but not badly enough to get banned) and the inability to distinguish the “core members” of a community from the “drive-by” posters (using Clay’s terminology) inevitably leads to protracted flame wars and nonsense threads. If people have some reason, no matter how slight, to protect their reputation, they’ll behave much more reasonably.