Peering into the Linux Kernel with Trace

Summary

Recently, I was working on a patch for a popular open-source project, and discovered that the test suite was failing intermittently. A closer look revealed that the last access time for some files in the project folder were changing unexpectedly, and this was causing a test to fail. (The failing test was not related to my patch.) Looking at the project code, it seemed impossible for it to be unexpectedly accessing those files during the test in question. Running the test case under strace confirmed that this was not happening. But incontrovertibly, the access times were changing. Could another process on the same machine be reading those files? But why? Could it be a bug in the operating system? Were my tools lying to me? Faced with a puzzle like this, the inclination might be to shrug one’s shoulders and forget about it, perhaps with a dismissive remark about the general brokenness of most software. (I’ve done that many times.) Anyways, it wasn’t my code which was failing. And yet, it seemed prudent to clear up the mystery, rather than bumbling along and hoping that what I didn’t know wouldn’t hurt me. This seemed like a good opportunity to try out the BCC tools. This is a powerful suite for examining and monitoring Linux kernel activity in real-time. Support is built in to the kernel (starting from 4.1), so you can immediately investigate when a problem is occurring, without needing to install a special kernel or reboot with special boot parameters. One of the more than 100 utilities included in the BCC tools is trace. Using this program, one can monitor when any function in the kernel is called, what arguments it receives, what processes are causing those calls, and so on. Having trace is really like having a superpower. Of course, the argument(s) of interest might not just be integers or strings. They might be pointers to C structs, which might contain pointers to other structs, and so on… but trace still has you covered. If you point it to the appropriate C hea...