I have a dystopian view of parallel processing.
This great talk by by Art Unger is titled “Everything You Know About Parallel Programming is Wrong”.
I think it is worse than that.
I think that Everything You Know About Parallel Programming is right, for some selected applications.
More accurately each of the many proposed parallel paradigms dominate the field for some set of real applications.
Further the paradigms typically have little in common.
The art of harnessing many computers is still centrifugal and that is probably good.
Some degree of bringing some commonality between these paradigms will be necessary for lack of researchers and implementers.
Even CPUs that go well past the point of diminishing returns for performance, are cheap!
The CPU is getting so cheap that we feel compelled to have many of them.
Parallel software and system architectures are intertwined and the specific examples that I have seen of these each fit some small part of the application landscape.
The first step in this direction, in the IBM lineage, was the 709’s ability to overlap computing and I/O, but only if the software logic was so adapted.
Long running production programs were usually able to exploit this.
Multiprogramming was able to exploit this parallelism without impinging on the design of most programs.
First came the vector machines and a few applications saw immediate speedup.
Other applications were eventually profitably adapted.
The von Neumann computer drove several generations of computer languages with largely the same expressive power.
The art of computer programming is largely devoted to single thread algorithms.