On Ten Year Trends - Parallel Goes Mainstream

Several bloggers have been discussing the current "ten year trends" that are transforming the computing environment, including Cloud Computing, Virtualization, Flash, and Mobility.

However, there is a deeper trend that while well underway, is so significant that it is often overlooked. That is the forced transition to loosely coupled distributed computing models caused by the knee in increasing compute performance by making individual processors faster.

Over the next five years, I believe we are going to see a full-fledged transition in hardware systems from tightly coupled single and multi-processor shared memory systems to loosely coupled many-processor systems that communicate via message passing. This trend is already evident in the high-performance computing space, where having hit the limits of individual processor performance, then having hit the limits of shared memory, virtually all new architectures have adopted this approach.

One of the fundamental drivers of this transition is a new economic model that measures computing power per dollar, or computing power per watt. This model favours many smaller less powerful processors over fewer high-power processors. After all, with Atom and ARM processors costing only a few dollars per core in volume, and performing around the same as a 1 GHz Pentium did just a few years ago, with a fraction of the power and heat dissipation, I predict we will soon see a transition to inexpensive servers where one or two rack units are crammed full of hundreds to thousands of these class of processors, all connected together by what looks like a Ethernet fabric.

The timing is right — Microsoft's talking about it, IBM's already done it, and the mobile market has brought the component volumes up to the point where the economics are right.

But five years does not make a ten-year trend: The larger change is that to programming models, languages and mindsets. Very few people in the industry can work comfortably and productively in massively parallel environments, as we have seen in the difficulty in building scalable web-based systems like Twitter. Computer Science still teaches programming and computer architecture around the old model of the "one true core", with parallel and distributed programming taught as a speciality, often at the graduate studies level.

It is going to take five or more years for these new massively parallel hardware and ways of thinking to filter down to the next generation of programmers who can think parallel, create new tools, libraries and stacks, and then play tens to hundreds of thousands of nodes like an instrument.

Imagine having a hundred thousand cores on your desktop. Imagine having ten million of them in your data centre. The problems are significant, and exciting.