Light Sleep: Waking VMs in 200ms with eBPF and snapshots

Summary

At Koyeb, we run a serverless platform for deploying production-grade applications on high-performance infrastructure—GPUs, CPUs, and accelerators. You push code or containers; we handle everything from build to global deployment, running workloads in secure, lightweight virtual machines on bare-metal servers around the world. Last week, we announced a major milestone in the ongoing journey of optimizing efficiency and cold starts: Light Sleep, which reduces cold starts to around 200ms for CPU workloads. To get there, during the last months, we had to rethink how we deploy apps, rewire how the network stack behaves when nothing is running, add snapshotting, lean heavily on some eBPF tricks, and work around a bunch of uncommon issues along the way. (Spoiler: networking is hell.) In this post, we’ll walk through our journey to bring 200ms cold starts on Koyeb — from the exact problems we hit, to why eBPF became our secret weapon, and how we got Scale-to-Zero working with near-zero cold starts. What we started with (and why it wasn’t enough) If you’ve followed us for a while, you probably know we used to rely on Firecracker: a lightweight virtual Machine Monitor (VMM) created by AWS that runs on top of the Linux kernel. It blends the security of traditional VMs with the speed and efficiency of containers. Firecracker is an excellent fit for multi-tenant environments because it's small, fast, and safe. But Firecracker comes with a few limitations, specifically around PCI passthrough and GPU virtualization, which prevented Firecracker from working with GPU Instances. At Koyeb, we provide a unified, hardware-agnostic experience to deploy across CPUs, GPUs, and accelerators. To keep our infrastructure consistent and flexible, we needed a virtualization layer that works across all workloads. That's why we moved away from Firecracker in favor of Cloud Hypervisor. Cloud Hypervisor is a modern VMM with broader hardware support and a more modern codebase. But as is usually the ...