TernFS – an exabyte scale, multi-region distributed filesystem

Summary

TernFS — an exabyte scale, multi-region distributed filesystem September 2025 XTX is an algorithmic trading firm: it builds statistical models that produce price forecasts for over 50,000 financial instruments worldwide. We use those forecasts to make trades. As XTX's research efforts to build better models ramped up, the demand for resources kept increasing. The firm started out with a couple of desktops and an NFS server, and 10 years later ended up with tens of thousands of high-end GPUs, hundreds of thousands of CPUs, and hundreds of petabytes of storage. As compute grew, storage struggled to keep up. We rapidly outgrew NFS first and existing open-source and commercial filesystems later. After evaluating a variety of third-party solutions, we made the decision to implement our own filesystem, which we called TernFS. We have decided to open source our efforts: TernFS is available as free software on our public GitHub. This post motivates TernFS, explains its high-level architecture, and then explores some key implementation details. If you just want to spin up a local TernFS cluster, head to the README. Another filesystem? There's a reason why every major tech company has developed its own distributed filesystem — they're crucial to running large-scale compute efforts, and liable to cause intense disruption if they malfunction. XTX was in the same position, so we designed TernFS to be a one-stop solution for most of our storage needs, going from relatively 'cold' storage of raw market data to short-lived random-access data used to communicate between GPU jobs running on our cluster. TernFS: Is designed to scale up to tens of exabytes, trillions of files, millions of concurrent clients. Stores file contents redundantly to protect against drive failures. Has no single point of failure in its metadata services. Supports file snapshot to protect against accidental file deletion. Can span across multiple regions. Is hardware agnostic and uses TCP/IP to communicate. Ut...