The race to build a distributed GPU runtime

Summary

For a decade, GPUs have delivered breathtaking data processing speedups. However, data is growing far beyond the capacity of a single GPU server. When your work drifts beyond GPU local memory or VRAM (e.g., HBM and GDDR), hidden costs of inefficiencies show up: spilling to host, shuffling over networks, and idling accelerators. Before jumping straight into the latest distributed computing effort underway at NVIDIA and AMD, let’s quickly level set on what distributed computing is, how it works, and why it's hard.Distributed computing and runtimes on GPUsDistributed computing coordinates computational tasks across datacenters and server clusters with GPUs, CPUs, memory tiers, storage, and networks to execute a single job faster or at a larger scale than any one node can handle. When a single server can’t hold or process your data, you split the work across several servers and run pieces in parallel, and if the job requires true distributed algorithms, not just trivially parallelizable independent tasks, then performant data movement is mandatory. Datasets and models have outgrown a single GPU’s memory. Once that happens, speed is limited less by raw compute and more by how fast you can move data between GPUs, CPUs, storage, and the network. In other words, at a datacenter scale, the bottleneck is data movement, not FLOPS.A distributed runtime is the system software that makes a cluster behave like one computer. It plans the job, decides where each piece of work should run, and moves data so GPUs don’t sit idle. A good runtime places tasks where the data already is (or soon will be), overlaps compute with I/O so kernels keep running while bytes are fetched, chooses efficient paths for those bytes (NVLink, InfiniBand/RDMA, Ethernet; compressed or not), manages multiple memory tiers on purpose (GPU memory, pinned host RAM, NVMe, object storage), and keeps throughput steady even when some workers slow down or fail.This is hard because real datasets are skewed, so a few pa...