The Limits of NTP Accuracy on Linux

Summary

Lately I’ve been trying to find (and understand) the limits of time syncing between Linux systems. How accurate can you get? What does it take to get that? And what things can easily add measurable amounts of time error? After most of a month (!), I’m starting to understand things. This is kind of a follow-on to a previous post, where I walked through my setup and goals, plus another post where I discussed time syncing in general. I’m trying to get the clocks on a bunch of Linux systems on my network synced as closely as possible so I can trust the timestamps on distributed tracing records that occur on different systems. My local network round-trip times are in the 20–30 microsecond (μS) range and I’d like clocks to be less than 1 RTT apart from each other. Ideally, they’d be within 1 μS, but 10 μS is fine. It’s easy to fire up Chrony against a local GPSTechnically, GNSS, which covers multiple satellite-backed navigation systems, not just the US GPS system, but I’m going to keep saying “GPS” for short. -backed time source and see it claim to be within X nanoseconds of GPS, but it’s tricky to figure out if Chrony is right or not. Especially once it’s claiming to be more accurate than the network’s round-trip time20 μS or so. , the amount of time needed for a single CPU cache miss50-ish nanoseconds. , or even the amount of time that light would take to span the gap between the server and the time source.About 5 ns per meter. I’ve spent way too much time over the past month digging into time, and specifically the limits of what you can accomplish with Linux, Chrony, and GPS. I’ll walk through all of that here eventually, but let me spoil the conclusion and give some limits: GPSes don’t return perfect time. I routinely see up to 200 ns differences between the 3 GPSes on my desk when viewing their output on an oscilloscope. The time gap between the 3 sources varies every second, and it’s rare to see all three within 20 ns of each other. Even the best GPS timing modules ...