NCCapital

On November 18, 2025, at 11:20 UTC, Cloudflare experienced a major network outage that prevented delivery of core traffic, resulting in error pages for users accessing customer sites. The issue stemmed from a permissions change in a database system, causing it to output duplicate entries into a feature file used by the Bot Management system. This doubled the file's size, exceeding software limits and triggering failures across the network. Initially mistaken for a DDoS attack, the problem was identified and mitigated by stopping the bad file's propagation and replacing it with a previous version, restoring core traffic by 14:30 UTC and full functionality by 17:06 UTC. Cloudflare apologized for the disruption, emphasizing its unacceptability given their role in the Internet ecosystem.

The outage manifested as spikes in HTTP 5xx errors, with unusual fluctuations where the system would temporarily recover before failing again. This was due to the feature file being regenerated every five minutes on a gradually updating ClickHouse database cluster, producing either good or bad configurations depending on which nodes were queried. As more nodes adopted the updated permissions, bad files became predominant, stabilizing the failure state. Resolution involved halting generation of the faulty file, inserting a known good version, and restarting the core proxy, with lingering issues addressed through service restarts.

Several services were impacted, including core CDN and security features returning 5xx errors, Turnstile failing to load, and Workers KV experiencing elevated errors due to core proxy failures. The dashboard remained mostly operational but login was hindered by Turnstile's unavailability. Email Security saw reduced spam detection accuracy and some Auto Move failures, though email processing continued unaffected. Cloudflare Access suffered widespread authentication issues for new sessions, with existing ones intact, and configuration updates delayed or failed. Additionally, CDN latency increased from CPU consumption by debugging systems.

Cloudflare's request processing involves termination at HTTP/TLS layers, routing through the core proxy (FL), and handling by Pingora for caching or origin fetches. The proxy applies customer configurations via modules, with Bot Management using a machine learning model fed by a periodically refreshed feature file. A query change caused duplicates, oversized the file, and triggered errors in the bots module. This affected both legacy FL and new FL2 proxy versions differently: FL2 showed 5xx errors, while FL generated incorrect bot scores of zero, leading to false positives in bot-blocking rules. A coincidental status page downtime initially suggested an attack.

The root cause was a ClickHouse query behavior shift from implicit to explicit access grants for underlying tables, exposing metadata from the 'r0' database and duplicating columns in unfiltered queries. This inflated the feature file beyond the Bot Management module's preallocated memory limit of 200 features, causing a panic in the Rust code and subsequent 5xx errors. Other systems like Workers KV and Access were mitigated by bypassing the core proxy. The dashboard and Turnstile impacts were resolved in phases, with full recovery by 15:30 UTC for Turnstile.