There's a theme I keep coming back to lately, of the relationship between hardware and software and how that relates to recent industry developments such as the acquisition of EMC by Dell. It has shown up for me on The Register, on StorageMojo, and last night on Twitter, so it seems like a good time to get these ideas out of my head by writing a blog post. Yeah, you can call that garbage collection if you want. ;)
When you need to set up some storage, you have to figure out where the software's coming from.
Bundled with hardware.
Bundled with a service.
Roll your own.
For the last two, you then need to decide where the software's going to run - your own hardware vs. someone else's (in the cloud). That's really six choices, or eight if you consider proprietary vs. open-source software to be separate categories.
Obviously, some of these choices make more sense than others. Historically, enterprise types have considered only #1 to have sufficient functionality and/or performance, while #4 wasn't even feasible. That's where EMC and NetApp and the rest of the storage rogues' gallery made their fortunes. However, all of that is changing. Standalone software, such as I work on, has become far more competitive. I won't say it's all the way to where it needs to be. Since I'm so involved in trying to get us there, very few indeed are as aware as I am of the distance still to be traversed. Still, we've come a long way and we have the momentum. The service offerings, often leveraging the open-source ones either directly or indirectly, have also become more competitive. I'm going to leave "roll your own" for later, but the point is that the "storage monolith" approach is under siege from multiple directions. Those markets are no longer captive. Those who are willing to do without a particular feature, or who can fill the gap with some niche software product, will leap at the chance. Both volume and margins for the big boxes will keep crashing down.
That's where the title of this post comes in. "Software will eat the world" is almost as popular as "disruption disruption disruption" was until companies like Uber and Theranos gave it a bad name. The thing is, storage hardware as a business is not going away. For one thing, someone obviously has to make all of the components. Seagate and WD/HGST (plus SanDisk now) are still going to make stuff and make money. For another thing, even storage systems won't be going away as a business. Storage workloads and power/density requirements create different design points than are well served by "standard" hardware. There's a different balance between CPU, cache, memory, internal and external interconnects, and other components (e.g. various NVRAM possibilities) to go with the masses of disk or flash. There will continue to be a market for hardware that's specially designed, chosen, assembled and/or tuned to serve those purposes. However...
It will be a smaller market than at present.
It can be just as well served by current compute-hardware vendors as by current storage-hardware vendors.
It will increasingly not run the hardware vendor's own software.
Yes, folks, the EMCs of the world might have to play nice with the Red Hats of the world to make their stuff run well together. I'm sure that will be lots of fun in lots of ways. Most relevantly, a smaller market plus a smaller share of that market plus sharing more of the wealth with partners isn't going to be good for their bottom line. Software will not eat the world, but it will definitely eat some large land masses.
OK, now here's the part I'm less happy about. The hardware vendors aren't the only ones under siege. We software vendors are too. As I pointed out to Chris Mellor at El Reg, the service providers are both an opportunity and a threat. Some people license our software to run in the cloud. Others use various cloud facilities instead of our software. (No, I don't care about the persistent rumors that e.g. Amazon's code is actually someone else's open-source code. Even if I gave those rumors any credence, it wouldn't matter much in the grand scheme of things.) Over time, though, I think the "threat" part will outweigh the "opportunity" part. Besides its convenience, the cloud pricing model makes that option appear cheaper than a software license, exerting downward pressure on prices.
Then there's the worst threat to us all, which is roll-your-own. Told you I'd get back to that. Rolling your own storage software used to be almost unthinkable. Now, not so much. Google, Facebook, Twitter, LinkedIn, and others have each rolled their own multiple times to address different use cases. They invented some of the techniques that are now well known among storage developers. I even like to think that I played some part in making them well known, through my articles and talks. As I said to Robin at StorageMojo, what's important is that the knowledge is out there. So are the components. Over time, it won't just be the big ubergeek companies rolling their own. More companies will start doing it, probably not from scratch but combining various open-source pieces with some of their own. "Mass customization" is not an oxymoron; it's the natural child of commoditization (for parts) and automation (to create systems).
Good morning, madam. What kind of storage system would you like me to build for you today?
Scary thought. That means that selling storage products is going to be hard for all of us. We'll be selling components, both hardware and software, or we'll be selling integration and support services. Somebody will always pay to have somebody else assemble the parts, maybe add some light customization, and support the result. There's a nice living to be made there . . . but no empires. I think that's what's behind some of the M&A activity. People can see the empires crumbling. Even if they haven't thought it through as consciously as I've tried to lay it out above, something in their gut tells them they'd better get what they can when they can. Everyone's rushing to build and defend their own little fiefdom before it all falls apart. Think about that the next time you see another storage company disappear. It won't be long.