Pdeathsig is almost never what you want

Summary

PDEATHSIG is almost never what you wantIt was a fine Sunday evening. I had just landed in LA for our company offsite, and I met the whole team in person for the first time. A little later, my phone buzzed with a notification, “Antonio assigned you an issue on Linear.” Antonio is an engineering team lead at Recall.ai, and he tasked me with optimizing Output Media start latency so that our customers’ AI agents would launch faster. I thought it was going to be the quickest, most straightforward thing I’d work on all week, but nope, it took much longer than anticipated. This is the story of how a seemingly simple task turned into a deep dive into the undocumented intricacies of the Linux kernel, Bubblewrap sandboxing, and Rust Tokio’s threading model.The planOutput Media is a Recall.ai feature that enables outputting ultra-low-latency audio and video from bots. Our customers use Output Media to build in-meeting AI agents, interactive applications, and more.Under the hood, Output Media works by rendering a customer-supplied web page into audio and video, which is then emitted through the bot.Because rendering a webpage involves running arbitrary untrusted code, we take a lot of measures to ensure this feature is secure. The most prominent technique we use is sandboxing. For performance and simplicity, we sandbox our Output Media code with Bubblewrap, giving it limited permissions – just enough for it to function properly.The current state of the feature was pretty decent – it was reliable and the number of things you could build with it was endless. There was just one small issue. Whenever customers activated Output Media, it took a sluggish 12 seconds for video to start streaming. Yikes! The reason this was taking so long was because when Output Media was activated, we’d launch an instance of Chromium to render the web page. Chromium is known to be complex, large and resource intensive, and the bots operate in a very resource-constrained environment.So, what could be do...