How fast are Linux pipes anyway?

Summary

2022-06-01 How fast are Linux pipes anyway? In this post, we will explore how Unix pipes are implemented in Linux by iteratively optimizing a test program that writes and reads data through a pipe. We will begin with a simple program with a throughput of around 3.5GiB/s, and improve its performance twentyfold. The improvements will be informed by profiling the program using Linux’s perf tooling. The code is available on GitHub. Chart showing the performance of our pipe test programs. The post was inspired by reading a highly optimized FizzBuzz program, which pushes output to a pipe at a rate of ~35GiB/s on my laptop. Our first goal will be to match that speed, explaining every step as we go along. We’ll also add an additional performance-improving measure, which is not needed in FizzBuzz since the bottleneck is actually computing the output, not IO, at least on my machine. We will proceed as follows: A first slow version of our pipe test bench; How pipes are implemented internally, and why writing and reading from them is slow; How the vmsplice and splice syscalls let us get around some (but not all!) of the slowness; A description of Linux paging, leading up to a faster version using huge pages; The final optimization, replacing polling with busy looping; Some closing thoughts. Section 4 is the heaviest on Linux kernel internals, so it might be interesting even if you’re familiar with the other topics treated in the post. For readers not familiar with the topics treated, only basic knowledge of C is assumed. Let’s begin! The challenge, and a slow first version # First of all, let’s start with measuring the performance of the fabled FizzBuzz program, following the rules laid down by the StackOverflow post: % ./fizzbuzz | pv >/dev/null 422GiB 0:00:16 [36.2GiB/s] pv is “pipe viewer”, a handy utility to measure the throughput of data flowing through a pipe. So fizzbuzz is producing output at a rate of 36GiB/s. fizzbuzz writes the output in blocks as big as the L2 cache...