Reverse Proxy Deep Dive: Why Load Balancing at Scale Is Hard

Summary

This post is part of a series. Part 1 - A deep dive into connection management challenges. Part 2 - The nuances of HTTP parsing and why it’s harder than it looks. Part 3 - The intricacies of service discovery. Part 4 - Why Load Balancing at Scale is Hard. Load Balancing One of the most critical roles for a reverse proxy is load balancing requests across different upstream hosts. From a list of upstream servers, the proxy must decide where each incoming request should go. The primary goals of load balancing are: Optimal resource utilization: Evenly distribute load to maintain consistent performance and avoid hotspots. Resilience: Ensure that the failure of a single server doesn’t cause outsized impact on capacity or latency. Operational simplicity: Simplify capacity planning, monitoring, and failover by evenly balancing load. Round-robin might work fine at small scale, but as systems grow, load balancing becomes far more complex due to many real-world challenges. What Makes Load Balancing Harder at Scale All Requests Are Not Equal While round robin sends an equal number of requests to each host, it doesn’t account for how different those requests may be. In large systems: Some requests are read-heavy, others write-heavy. Some carry large payloads in the request, others in the response. Some are CPU-bound while others are IO-bound. This disparity means that round robin can overload one server while leaving others underutilized. Example: Imagine two types of requests: image uploads (large payloads, heavy compute) and profile lookups (lightweight reads). Sending equal numbers of each to all hosts can cause unpredictable load patterns. Alternatives: Least Connections: Distributes requests to the host with the fewest active connections. This requires the proxy to maintain accurate connection tracking. Power of Two Choices (P2C): Randomly pick two hosts and send the request to the one with fewer active connections. A randomized algorithm but simple, fast, and surprisingly ...