Puzzle questions are useless in interviews. Yes, I know there’s a real cult around them, and I don’t deny that they ever have any value, but the conditions for their successful use occur so rarely that it’s not worth qualifying that original statement. In the vast majority of cases, puzzle questions are a way for incompetent interviewers to kill time instead of getting real information. To see why, let’s consider what has to happen for a puzzle question to provide actual value.

First, you have to pick a good question. Actually you have to pick more than one, in case the candidate already knows your first one. After all, there are whole sites devoted to collecting these, and people who have practically made a career out of solving puzzles instead of doing real work. For similar reasons, trick questions aren’t very useful either; you don’t learn much about a candidate who just gives you the answer because they’ve seen it before. The question has to be relatively straightforward, but difficult or complex enough that you’ll learn something from watching the candidate solve it . . . but not so difficult or complex that it consumes the entire interview. The question should be one for which the answer, or the path to the answer, is somewhat domain-relevant, but not so domain-specific that only you and your office buddies are likely to know it off the tops of your heads. Do you see yet how difficult it is to find a truly useful question?

Now you have to administer the question. Puzzle questions can’t be about testing technical knowledge. If you want to test technical knowledge, ask technical questions. I recently encountered a high-level consultant who didn’t even know what consistent hashing is, in a context where it’s pretty much a standard technique. Had it been an interview, I could have learned more by asking a candidate to define or explain consistent hashing than by asking any number of puzzle questions. Every field has such key terms or concepts. Figure out what yours are, and ask about them directly instead of playing games. Back to our point, the purpose of a puzzle question must be something else. Maybe it’s to test how the candidate combines ideas, or how well they work with someone else toward a solution. How many engineers are really able to evaluate such things? How many are able to remove themselves from their own preconceived idea of the solution method or result to conduct such an evaluation without injecting their own bias into the process? That’s exactly the kind of thing we’re known to do poorly. Someone with specific training in psychology or management can do it right, but the average engineer without such training has almost no chance.

So, by some miracle, you’ve been able to select a good question and administer it well. Now you have to evaluate the response. The question represents a subject you know well, that you hope the candidate doesn’t know as well, and you’re asking them to perform on the spot with you looking on critically. How many of us (engineers) are comfortable getting up at a whiteboard and doing even the most familiar things in front of strangers under time pressure? How many specifically write code under those conditions (since coding questions are the most common sub-genre)? Most take breaks to check the manual for the specifics of a call, go back and rewrite portions three times, insert five syntax errors in even a trivial code fragment, etc. Now, how many of us are capable of accounting for the fact that somebody else’s “inferior” performance is the result of something other than their being less intelligent than ourselves? Again, that’s a notorious weakness. In reality, most of us will systematically underestimate the candidate’s ability relative to our own, rejecting perfectly good programmers because they weren’t good trick ponies as well.

So, with all the pitfalls in selecting a question, administering it, and evaluating the results, how often do you think it happens that the puzzle question provides good value for the time spent? Almost never. Yes, it could happen, but you’re just about as likely to get struck by lightning. Counting on either event would be a poor basis for something as important as a hiring decision.

This being the internet, I’m sure someone will try to claim I’m just venting because I flubbed a puzzle question. Wrong. As far as I know, my performance on such questions has never prevented me from getting to the next round or getting an offer, and has often contributed positively to such outcomes. The one person who asked such questions this time around might well have been among those exceptional few who can actually avoid the pitfalls I’ve mentioned. I’m writing this more out of pique at my colleagues who I’ve seen waste valuable time with this silly gamesmanship, who left our group chronically understaffed due to the impossibility of anyone passing such a gauntlet. In consequence, I’ll offer this piece of advice not for interviewers but for candidates: any time an interviewer relies on a puzzle question, take the opportunity to re-evaluate whether they are good enough for you. The odds go down with every puzzle question you’re asked.