I came up with this little parable about coding in complex systems. Warning: it’s a bit graphic.

Imagine you have a remote view of an ordinary room, with doors at each end. You see a man enter at one end. He crosses to the other end, pausing a couple of times and once stooping to examine something on the floor, then exits at the other end. There seems to be nothing remarkable about this. Then you watch a second man try to cross the room. When he gets half-way across, panels in the walls slide down and the man is skewered with arrows from several hidden crossbows. The screen flickers. A third man tries to cross, and doesn’t even get a quarter of the way across before a green gas seeps from vents near the ceiling and he collapses. The screen flickers again. After you watch a score more of failed attempts to cross the room, the smoke from the latest explosion reveals something: the room is criss-crossed with beams of light, some of which turn on and off, and it’s when people cross these beams that Bad Things happen. You think back to the first man, and realize that he might crossed in the only possible safe way. You briefly consider that he was lucky, but it’s just too implausible. It’s much more likely that he knew exactly what he was doing, that his pauses and stooping were carefully calculated to avoid the various traps. Only then do you realize how skillful his performance was.

Yeah, I know, it’s like those “ninja” metaphors that I usually consider so childish. I was going to try something less colorful, but Amy happened to overhear this little story as I was trying it out on Cindy, and she keeps asking me to tell it again, so maybe being childish isn’t so bad. Anyway, the point is that working on a very complex system (such as I do on GlusterFS) can be a bit like crossing that room. The code contains all sorts of hidden traps for the unwary. I’ve taken my share of arrows in the knee, that’s for sure (and now you know what inspired the metaphor). Usually you have to go through many iterations of being that second or third or tenth person before you have a day where you feel like that first one. You know all of the non-obvious requirements and constraints (traps) between you and what you’re trying to accomplish. Maybe you even relax (disarm) a couple, but mostly you just do the delicate dance around them and get to the point where your new feature works (get out alive). Then somebody watching says, “Pffft, you just walked across a room. What’s hard about that?” That’s why people who have learned – the hard way – how to bend a piece of code in ways it wasn’t originally intended to bend sometimes seem intolerant of dabblers’ or onlookers’ commentary. That’s one of the reasons I stuck with this metaphor: the complexity is hidden. I thought of using a different metaphor of a complex mechanism with many levers and such, but anyone seeing such a mechanism would realize it’s complex. In this case, you could watch the first man cross the room a hundred times and never realize he was engaged in a complex task.

If there’s a moral here, it’s twofold. First, don’t assume something’s easy just because you see someone do it without mishap or obvious effort. They might be applying a great deal of hard-won knowledge to make things go so smoothly. Second, when you’re attempting to do something in a complex environment, expect that you’ll get killed a few times. Respect the dangers, but don’t be afraid of them. You can always reload from your last save. Each time you find a trap, you learn. After enough tries, you might find a way to avoid them all, and the result might even be more satisfying than if things had been easy to begin with.