Hacking Google Chrome Source Code: Make Puppeteer work over Redis PubSub

Summary

During my time at Pikkit, we frequently needed to spin up large numbers of browser sessions to automate tasks across various websites. The traditional approach is to run Chromium inside containers and expose a DevTools endpoint using the --remote-debugging flag. Architecturally, it looks like this: Puppeteer client (Devtools) ⇌ Chromium Remote Debugging Server (Devtools) The main challenge with this setup is that the Chromium Remote Debugging Server don't become available instantly, it takes time for the browser to start. This forces you to repeatedly poll for reachability, which introduce complexity and instability. Also, because DevTools communicates strictly over TCP/IP, any lost packets can instantly break the session. Which is an issue that happens on about 1% of the sessions in production. One way to solve this would be to use a reverse connection model. Instead of waiting for the client to connect to the Chromium Remote Debugging Server, we could make the Chromium Remote Debugging part connect to a central server. Or even better, a publish and subscribe stream through Redis, which would handle buffering for us: Puppeteer client (Devtools) ⇌ Puppeteer Server (Devtools) ⇌ Redis (Redis) ⇌ Chromium Remote Debugging Unfortunately, Chromium doesn’t support Redis, or any similar messaging layers, out of the box. Most setups work around this by introducing an additional intermediate layer that forward packets by connecting in between the Redis and Devtools. Puppeteer client (Devtools) ⇌ Puppeteer Server (Devtools) ⇌ Redis (Redis) ⇌ Redis to devtool translation ⇌ Chromium Remote Debugging While functional, this approach adds even more moving parts, which hurts both speed and reliability. So instead of layering on additional components, I want to take this a step further and integrate Redis support directly into Chromium’s DevTools server. Let's dive in. Chromium's Codebase Chromium’s codebase is enormous. Whenever I work with a project of this size, I start with a ful...