Exploring a Language Runtime with Bpftrace

Summary

So I have been having quite a bit of fun learning about eBPF. It’s been on my todo list for like two or three years, but I’ve finally made the time to actually start to figure it out, and have already found some neat stuff, and had to do some fun hacking. Let’s set the stage. eBPF is an instrumentation and tracing system built into Linux (and Windows these days!). The general thrust of it is that you provide the kernel with some bytecode, which it verifies, then executes in kernel context. You’re able to collect all sorts of information via this bytecode, which then can be dumped out. There are multiple ways to get BPF bytecode into the kernel, but I’m going to talk about bpftrace, which provides an awk like language for writing BPF programs. Now, there’s a fair amount of reasonably comprehensive documentation about bpftrace so I’ll give only the tiniest intro, and then get into some of the problems I’ve been curious about and the solutions I’ve built around BPF to get answers. Your basic bpftrace program looks like this: some:probe /* Probe */ { /* Action List */ @start[tid] = nsec } some:other:probe /@start[tid]/ /* filtered by having a start timestamp with this tid that is nonzero */ { $duration = nsec - @start[tid] print($duration) delete(@start[tid]) // remove the key entry so we can get the next duration. } A couple of things worth noting in this sketch: Variables that start with @ are maps, associative keyed dictionaries. These are global. Variables that start with $ are locals to an action set. /something/ is a filter applied on a probe There are many builtin functions (print, delete) and ‘getters’ like nsec which returns a timestamp and tid which returns the thread id. Rooted profiling SpiderMonkey is a precisely rooted JS engine, so we have a Rooted type, that informs the GC about C++ object pointers. One thing we've noticed is that occasionally we see the Rooted constructor in profiles; yet it's also fast enough that it doesn't show up often. We also have...