Memory is slow, Disk is fast – Part 1

Summary

TL;DR Hardware got wider, not faster. More cores, more bandwidth, huge vector units — but clocks, IPC, and latency flatlined. Old rules like “memory is faster than disk” are breaking. To go fast today, you have to play the new game. “CPUs keep getting faster every generation” Over the past 20 years or so computer hardware has evolved such that some facts we “know” about computers are wrong. Even among computer scientists, or perhaps especially among computer scientists, intuitions are off target. Let's consider a simple statement: “CPUs keep getting faster every generation” This is obviously not wrong entirely. Computers are certainly getting faster and more capable. When we put some hard numbers to this we can see a lot of things have been and continue to be improving exponentially. What is scaling? If we look at the promotional materials for CPUs, you can see the product specs are clearly improving. For example: Transistor Count As much as there is hand wringing about the death of Moores Law, we are producing CPUs with ever increasing transistor counts. Is this driven by more silicon being packaged in increasing sophisticated ways and moving into the Z axis rather than the core feature size shrinking of the past? Are we doubling slower than in the past? Who cares? Produc Core Counts The logic transistors that dominate the size of a core on a chip scale well with the lithography changes. This makes scaling CPU core counts cut and paste easy. Vector Operations For vectorized operations underlying graphics, physics and AI calculations processors have been innovating at a rapid rate. Although compilers don’t always find potential optimizations, the industry has done a good job of having support for new vector instructions in compilers and intrinsics in sync with release of the hardware. The performance gains for implementing the latest vector calcs can be dramatic. Memory Bandwidth The industry has been very good at cooperatively developing and adopting new standards ...