Behind the scenes: Redpanda Cloud's response to the GCP outage

Summary

On Jun 12th, 2025, Google Cloud Platform (GCP) experienced an unfortunate global outage triggered by an automated quota update to their API management system. What was a major outage for a large part of the internet was just another normal day for Redpanda Cloud customers. While GCP dealt with the widespread disruption that impacted many critical services, Redpanda Cloud clusters in GCP remained stable, thanks to being purposely designed for the SLA we offer, along with a cell-based architecture that we also made a product principle. But behind the scenes, it was far from quiet. This post provides a brief timeline of events from our own experience, our response, previously untold details about Redpanda Cloud, and closing thoughts on safety and reliability practices in our industry.Why do incidents like this happenModern computer systems are complex systems — and complex systems are characterized by their non-linear nature, which means that observed changes in an output are not proportional to the change in the input. This concept is also known in chaos theory as the butterfly effect, or in systems thinking, with the expression, “The whole is greater than the sum of its parts”. When this mathematical fact is acknowledged, safety and reliabiilty measures are put in place, such as closing feedback control loops, phasing change rollouts, shedding load, applying backpressure, randomizing retries, and defining incident response processes, among others.GCP’s seemingly innocuous automated quota update triggered a butterfly effect that no human could have predicted, affecting several companies — some known for their impressive engineering culture and considered internet pillars for their long-standing availability record.The timeline18:41:00 UTCOur Google Cloud Technical Account Manager (TAM) notified us about the outage:18:42:00 UTCWe began to assess the impact on our Redpanda Cloud GCP customers, including whether we had received any support tickets. 18:43:00 UTCWe noticed...