Cloudflare outage should not have happened

Summary

Cloudflare outage should not have happened, and they seem to be missing the point on how to avoid it in the future November 26, 2025 by Eduardo Bellani Yet again, another global IT outage happen (deja vu strikes again in our industry). This time at cloudflare(Prince 2025). Again, taking down large swats of the internet with it(Booth 2025). And yes, like my previous analysis of the GCP and CrowdStrike’s outages, this post critiques Cloudflare’s root cause analysis (RCA), which — despite providing a great overview of what happened — misses the real lesson. Here’s the key section of their RCA: Unfortunately, there were assumptions made in the past, that the list of columns returned by a query like this would only include the “default” database: SELECT name, type FROM system.columns WHERE table = ‘http_requests_features’ order by name; Note how the query does not filter for the database name. With us gradually rolling out the explicit grants to users of a given ClickHouse cluster, after the change at 11:05 the query above started returning “duplicates” of columns because those were for underlying tables stored in the r0 database. This, unfortunately, was the type of query that was performed by the Bot Management feature file generation logic to construct each input “feature” for the file mentioned at the beginning of this section. The query above would return a table of columns like the one displayed (simplified example): However, as part of the additional permissions that were granted to the user, the response now contained all the metadata of the r0 schema effectively more than doubling the rows in the response ultimately affecting the number of rows (i.e. features) in the final file output. A central database query didn’t have the right constraints to express business rules. Not only it missed the database name, but it clearly needs a distinct and a limit, since these seem to be crucial business rules. So, a new underlying security work manifested the (unintended) po...