A database permissions change doubled a config file, and Cloudflare's proxy fell over
Happened ·Published
Cloudflare's network started failing at 11:20 UTC, and core traffic was largely flowing again by 14:30. It was not caused by a cyber attack or any malicious activity. A change to database permissions made a generated file double in size, past a limit in the software that reads it. Cloudflare called it its worst outage since 2019.
What users saw
- Error pages (HTTP 5xx) on sites using Cloudflare's CDN and security services, for traffic that depended on its Bot Management module.
- Turnstile failed to load. The Cloudflare dashboard itself was mostly up, but most users without an active session couldn't log in, because Turnstile was unavailable on the login page.
- Workers KV returned significantly elevated 5xx errors. Cloudflare Access logins failed for most users from the start until 13:05; existing Access sessions were unaffected.
- Significantly slower responses from the CDN during the impact period.
- Cloudflare's status page also went down, though it's hosted entirely off Cloudflare's infrastructure. It turned out to be a coincidence, but it led some of the team to suspect an attack.
What broke
- 11:05 UTCPermissions changeA ClickHouse change made users' access to underlying tables explicit. A metadata query that didn't filter by database started returning duplicate columns.
- EVERY 5 MINFeature file doublesThe Bot Management feature file is built from that query every five minutes. It filled with duplicate rows and doubled in size.
- LIMIT 200Limit exceededThe proxy preallocates memory for up to 200 features; normal use is about 60. The bad file had more than 200.
- RESULTProxy panicsIn the new FL2 proxy the limit check raised an unhandled error, the proxy panicked, and requests got 5xx errors. The older FL proxy returned no errors, but gave every request a bot score of zero: customers with rules blocking bots would have seen large numbers of false positives, while those not using bot scores saw no impact.
Why it kept recovering and failing again
The file was rebuilt every five minutes, and only the parts of the database cluster that had already received the permissions change produced the bad version. So every five minutes the network got either a good file or a bad one, recovering and failing again, until every ClickHouse node was generating the bad file and it stayed broken.
Cloudflare says this pattern initially led it to believe it might be under attack.
Timeline (UTC)
- Database access control change deployed.
- Impact starts: first errors on customer HTTP traffic.
- First automated test detects the issue; manual investigation starts at 11:32 and the incident call at 11:35.
- Workers KV and Cloudflare Access switched to an older version of the core proxy, where the same issue had a smaller impact; impact reduced.
- Creation and propagation of new Bot Management files stopped; a test with the old file succeeds.
- Resolution begins: a known-good file is inserted into the distribution queue and the core proxy restarted; a correct file is deployed globally and most services start operating correctly.
- All services resolved.
Cloudflare's post gives two start times: 11:20 in its summary and 11:28 in its timeline table.
The networking lesson
- Generated config is still input. Validate files your own systems produce the way you'd validate a customer's. Cloudflare lists this first among its fixes.
- One push, every machine. The file was rapidly propagated to every machine in the network. Cloudflare's fixes include more global kill switches for features.
- Limit the blast radius. Staged rollouts, where a change reaches a few machines before all of them, are another way to contain a bad file. our analysis
- Fail soft, and check what the soft failure does. A quieter failure isn't automatically a safe one: a wrong score can still act on traffic, as the older proxy showed. our analysis
- A rhythm is a clue. When a failure comes and goes on a fixed interval, look for what runs on that interval before assuming an attacker. our analysis
A generator script builds one config file and pushes it to every router at once. Make the generator emit duplicates, watch every node fail together on the same five-minute rhythm, then fix it with a validation gate and a last-known-good rollback.
Source: Cloudflare outage on November 18, 2025, Cloudflare (official post-mortem). Facts, times and causes are as the source states them; anything marked our analysis is ours.