One part of my job has to do with sales. Historically this has been mostly post-sales, but I’m increasingly – and welcomely – being consulted about storage configurations during the pre-sales phase as well. I often find myself having to be quite aggressive in rejecting underprovisioned or unqualified solutions, because of a very simple fact:
Customers are often ignorant or unreasonable, and it’s my job to be a counterbalance.
When I say that customers are ignorant or unreasonable, I mean no insult. Storage is probably the least commonly understood part of most systems. The people who buy systems tend to understand hardware and application software fairly well, networking a little, and storage least of all. A parallel filesystem, which combines storage and networking and parallel/distributed programming, is something most people are simply unable to conceptualize properly. Customers have their own problems to solve. If they don’t understand ours, even when ours affect how they solve theirs, they simply need to be educated.
Customers being unreasonable, though, is a different problem. The customer we see is often not the user, but rather some IT/procurement professional. Nonetheless, their primary duty is to their users and they serve that duty by demanding maximum performance for minimum cost. They might know that storage often accounts for 30-60% of total system cost in commercial environments, or 10-20% in HPC, but they’ll still try to bargain that down aggressively. Initial demands for storage performance commensurate with compute performance for only 5% of total system cost are not uncommon, and I’ve even seen 1%. So, what happens if I come back with a configuration that’s more like 15%? The salespeople, who are in the middle of all this, then get stuck in the technical equivalent of the Overton Window in politics. They see one side offering an extreme position and the other offering a moderate one, leaving them with a range of choices that’s all on one side of the true median. If they try to compromise by splitting the difference, they’re likely to end up with a solution that increases support costs and leaves customers with a bad impression.
So, what’s the solution? Well, if the customer is going to be an aggressive advocate for maximum performance and minimum cost, then I must be the exact opposite. My proposal should therefore not be a 50th-percentile best guess, but a 99th-percentile solution involving the most hardware in only the most thoroughly tested configurations. It’s like our system of justice, in which the prosecution and defense are not only expected but required to offer the strongest possible advocacy for opposing positions so long as they adhere to the rules and the facts. Only by being an absolute “Dr. No” when it comes to evaluating proposed configurations can I, or anyone in an equivalent position, expect the end result to be a configuration that both satisfies the customer and avoids costly support interventions later.