OOMProf: Profiling on the Brink

Summary

It was just a little while past the Sunset Strip They found the girl's body in an open pit Her mouth was sewn shut, but her eyes were still wide Gazing through the fog to the other side "Black River Killer" by Blitzen Trapper Introduction This one's personal! For 15 years working on DBMS systems the OOM killer has led to more than its fair share of debugging rabbit holes. Anyone who's been around the block in Linux systems programming has probably crossed paths with the Linux OOM killer. This is the part of the kernel that attempts to maintain forward progress when faced with the impossible situation of applications wanting to use more memory than the system has. The OOM killer balances a lot of competing interests but in the end, it just picks a victim process and kills it. You're a busy professional so the TLDR is we built an OOM monitoring system called OOMProf in eBPF that profiles Go programs at the point they are OOM kill'd capturing allocations up to the bitter end to give developers a better idea of exactly what went wrong. If you're lazy or unprofessional or just want to know more about how the sausage was made read on! And as an extra bonus we've litered this blog post with stanzas from a great modern folk murder ballad for you to enjoy! They booked me on a whim and threw me deep in jail With no bail, sitting silent on a rusty pail Just gazing at the marks on the opposite wall Remembering the music of my lover's call The Problem With OOM Kills The problem with OOM killed programs is that developers who are faced with explaining what happened have very little to go on. First of all the application that leads to the OOM killer firing may not be the process that gets killed. And in the more common case where the application getting killed is actually the one that did something to deserve being killed the straw that breaks the camels back may be some innocuous common allocation or even some memory allocator induced page fault and unrelated to the actual pile u...