The Evolution of Caching Libraries in Go

Summary

The Evolution of Caching Libraries in Go For the last few years, I've been developing caching library, and today I'd like to talk about the evolution of caches in Go and where we stand today. Especially since Go 1.24 was officially supposed to focus on caching improvements, but I haven't heard much news about them - time to fix that . On-heap VS Off-heap Before we begin, I believe it’s important to mention that in programming languages with GC (such as Go), caching libraries are divided into two main types: on-heap and off-heap. On-heap caches allocate memory in the heap, while off-heap caches allocate memory (for example) using mmap and then manage it manually. Off-heap caches offer two primary benefits: they eliminate GC pauses regardless of entry count, and they maintain minimal memory overhead for metadata. These advantages, however, are offset by several limitations: Poor eviction policy in most cases. Go off-heap caches often use something very similar to FIFO, which typically has a worse hit rate than LRU. You have to convert your keys and values into strings/byte slices, as these are used as keys and values. This conversion will likely be much more expensive than any in-memory cache access operation. Lack of many useful features, which we’ll discuss later. Off-heap caches are typically useful when your cache must keep all data in memory permanently and rarely (if ever) evict entries. This becomes essential, for example, when your service's SLA depends on maintaining a hit rate consistently above 99.9%. On-heap caches, in exchange for some GC overhead, solve all of the above drawbacks. I should mention right away that in this article, we’ll only discuss on-heap caches, as they evolve independently of off-heap caches—and my knowledge of off-heap cache internals is relatively superficial. Just keep these differences in mind when choosing a cache. Early development For a long time, Go lacked an advanced concurrent cache. All libraries essentially offered a mutex...