Not to contradict or anything — I’m not even sure if this could be better used as a PRO or CON argument for your point — I’ll just add this as something possibly (hopefully!) relevant:

http://lacos.hu/lbzip2-scaling/scaling.html

why didn’t the market reward SiCortex then?

The market seemed quite willing to; it’s the investors who failed. No matter how good a startup’s technology is, they face an uphill battle in the marketplace. Time after time, we’d hear prospective customers say they loved the product, they loved the people they’d worked with through the evaluation period, but they were afraid to spend that much for a system from a small company that might not survive. Their prophecy of failure turned out to be self-fulfilling, just as a prophecy of success would have been. If those same customers had evaluated products purely on the merits, many of them would have bought our systems and we would have been fine. Building up that trust requires demonstrating that the original technical success is repeatable, but building that second generation – which we were doing – takes more time and money that early-adopter customers put in – hence the need for later-stage investment. More here, and it was a similar story at Revivio and many other places. The lesson from SiCortex is not a technical one, but that startups should worry about their investors’ viability as much as the other way around.

Getting back to the topic, it’s important to note that all of the very biggest systems in the world – e.g. Roadrunner, Jaguar, Intrepid – are based on essentially the same kind of non-shared-memory structure as SiCortex. The Blue Gene and PowerXCell8i systems from IBM are particularly interesting in this context because they use explicit memory hierarchies even within nodes. Shared memory is great up to its scaling limits, but its scaling limits in the real world are determined by programmer behavior. That behavior is typically to abuse the sharing so that the amount of coherency traffic rises as (approximately) the square of the processor count, rapidly overwhelming even the interconnect resources that are available on-chip. So far the only viable solution has been to constrain use of the interconnect by adopting models like MPI. Hadoop manages to do something like that across larger systems that share storage rather than memory, where similar problems also occur. Perhaps something like Cilk or a distributed Grand Central Dispatch can do that for larger systems that share memory. It’s still an active area of research, and I wish those researchers well.

If you have a nice abstraction, like CISC, and its specification matches exactly what a user wants to do, then it’s more efficient than hand-called RISC, because the abstract->small-pieces translation happens at a lower level, thus less round-trips.

If the abstraction leaks or doesn’t match exactly what a user tries to do, then users will start to abuse the high-level API.

I think the example I’ve read about was GPU’s (mis)used for non-rendering, generic computational tasks, and conversely, high-performance general purpose CPU’s used for graphical rendering, for greater freedom of expression. CPU’s were too slow, thus a few primitives were crystallized and pushed down into hardware (GPU’s). Now, people started wanting to tap that vector architecture for other purposes (coveting an access to the lower levels of the GPU), while simultaneously finding the primitives too restrictive even for graphics, with powerful CPU’s being available.

I know about these topics only superficially as you can probably tell, but the conversation I seem to remember went something like this.

Second, the market is fashion driven and just plain unreasonable. If I get your point, you say that big SMP is disappearing rightfully, for the reasons (1) its complexity makes it too expensive in the market, and (2) programmers fail at programming it (the abstraction leaks and one should handle node distances explicitly, for example). This might be the case, but why didn’t the market reward SiCortex then? I may be a bit off here, but I believe SiCortex mainly supported MPI (which appears to be the correct approach with the current swing of “RISC” in parallel programming), and it would have been cheaper as well in the long run, both for being very energy-efficient and more useful to program (more explicit, but with less obscure bugs). Still, the market didn’t keep it alive.

Sorry, this is wildly incoherent, and I apologize for that. I can’t express it better for now, I just feel that the market is a fickle mistress and no phenomenon can be justified by it.