How we tracked down a Go 1.24 memory regression

Summary

When Go 1.24 was released in early 2025, we were eager to roll it out across our services. The headline feature—the new Swiss Tables map implementation—promised reduced CPU and memory overhead. Our story begins while the new version was being rolled out internally. Shortly after deploying it to one of our data-processing services, we noticed an unexpected memory usage increase: We observed the same pattern, a ~20% increase in memory usage, across multiple environments before pausing the rollout. To confirm our suspicions, we conducted a bisect in the staging environment, which pointed directly to the Go 1.24 upgrade as the culprit. But here's where things got truly puzzling: The increased memory usage wasn't showing up in Go runtime metrics and live heap profiles, which meant that, from the Go runtime's perspective, the service wasn't using more memory. This caught our attention immediately. After all, Go 1.24 was supposed to reduce memory usage thanks to Swiss Tables, not increase it. In this two-part series, we'll share how we investigated this surprising increase and how the same release ultimately helped us reduce our memory footprint. In this first post, we'll walk through how we diagnosed a subtle memory allocator regression introduced by a runtime refactor and how we worked with the Go team to identify and confirm the root cause. In Part 2: How Go 1.24's Swiss Tables saved us hundreds of gigabytes, we'll show how Go 1.24's new Swiss Tables implementation dramatically reduced the memory usage of a large in-memory map—yielding a net win across our highest traffic services. Before diving deeper, we needed to eliminate the most likely suspects. Go 1.24 introduced a couple of major changes that could potentially impact memory usage, so we systematically tested each one: Swiss Tables: This feature was supposed to reduce memory usage, but we needed to verify it wasn't somehow causing our problem. We created a test build with Swiss Tables disabled by setting the GOEX...