Today I Learned.

And, I actually prefer it that way.

I relearned the lesson of Amdahl's Law, and I learned a new word, "bathetic". No, I didn't spell that wrong. Bathetic, with a 'B'. Which, in a similar but not exactly the same vein as 'pathetic', draws on the Greek word 'bathos' meaning depth. But what it really means is an abrupt, absurd fall into mediocrity or chaos.

The basic lesson of Amdahl's Law is that parallel computing is hard. The slightly more than basic lesson is that adding additional simultaneous processing to a problem only helps in so far as there is simultaneity to be had.  Which is to say that if half of your work is completely serial and half is completely parallelizable, you can only expect to halve your run time with all the additional processors in the universe.

Today's lesson from Amdahl takes a bit of a jag. If the gods give you a magic computer that does the parallel portion of your calculation infinitely fast, effectively reducing its run time to zero, but it runs the serial portions of your code slower, then you should think carefully before making promises about what that gift can do. Even if this computer instantly returns the result of the parallel part, if it also doubles the run time of the serial part of the code, then the gift is a wash if half the work is serial and half the work is parallel. Fortunately, in the real world, the best scientists build codes where more than 99.999% of the work is parallelizable.

Unfortunately, this is, in some ways, the problem with GPUs, Xeon Phis, FPGAs, and other computing accelerators. They do some things extremely well, and while they don't reduce the parallel run times to zero, they may cut them by significant factors. But to do this, they have to sacrifice performance on serial execution. If you don't watch out for this, you can get killed by it, bathetically.