Mind the encryptionroot: How to save your data when ZFS loses its mind

Summary

I unfortunately did not keep track of every resource I consumed, but in addition to reading the source and docs, I found these talks by Jeff Bonwick, Bill Moore, and Matt Ahrens (the original creators of ZFS) to be particularly helpful in understanding the design and implementation of ZFS: I highly recommend watching them all despite their age and somewhat poor recording quality, but will summarize the relevant information for those who don't have 3 hours to spare. ZFS is a copy-on-write filesystem, which means that it does not overwrite blocks in place when a write is requested. Instead, the updated contents are written to a newly allocated block, and the old block is freed, which keeps the filesystem consistent if a write is interrupted. All blocks of both data and metadata are arranged in a Merkle tree structure where each block pointer contains a checksum of the child block, which allows ZFS to detect both block corruption and misdirected/phantom reads/writes. This means that any write will cause the block's checksum to change, which will then cause the parent block's checksum to change (since the parent block includes the block pointer which includes checksum of the child block that changed), and so on, all the way up to the root of the tree which ZFS calls an uberblock. Uberblocks are written atomically, and because of the Merkle tree structure, they always represent a consistent snapshot of the entire filesystem at a point in time. Writes are batched together into transaction groups identified by a monotonically increasing counter, and each transaction group when synced to disk produces a new uberblock and associated filesystem tree. Taking a snapshot is then as simple as saving an uberblock and not freeing any of the blocks it points to. In addition to the checksum, each block pointer also contains the transaction group id in which the child block was written, which is called the block's birth time or creation time. ZFS uses birth times to determine which bl...