Cloudflare outage on November 18, 2025

A database permissions change doubled a file and took down the network

At 11:20 UTC on 18 November 2025, Cloudflare's network began failing to deliver core traffic. Users hitting customer sites saw error pages indicating a fault inside Cloudflare's network. The cause was not an attack. A database permissions change caused a Bot Management feature file to double in size, and the software that routes traffic across the network could not handle the larger file.

The feature file is generated every five minutes by a query on a ClickHouse database cluster. The cluster was being gradually updated to improve permissions management, and the query only produced bad output on nodes that had already been updated. This meant that for a period, good and bad configuration files were alternately generated and propagated across the network, which caused the system to recover and then fail again at five-minute intervals. That periodic recovery made the outage look like something other than an internal configuration problem — initially, it was mistaken for a possible attack.

Once every ClickHouse node had been updated, the bad file was generated consistently and the network stayed in a failing state. The issue was correctly identified at 14:30, and the fix was to stop generation and propagation of the bad file, manually insert a known-good file into the distribution queue, and force a restart of the core proxy. Traffic largely recovered by 14:30, with a long tail of service restarts continuing until all systems returned to normal at 17:06.

During the impact window, the CDN also showed significantly higher latency alongside the HTTP 5xx errors. This was caused by debug and observability systems consuming large amounts of CPU while automatically enriching uncaught errors with extra diagnostic information.

Why a bigger file broke traffic routing

The feature file is read by the software that routes traffic across the network. That software keeps Bot Management current with evolving threats by reading the file, but it had a hard limit on file size that was lower than the doubled file produced by the misconfigured database query. When the file exceeded the limit, the routing software failed.

The chart below shows the volume of 5xx errors served across the network. The spike and subsequent fluctuations reflect the alternating good and bad file generations, and the long tail reflects the team restarting services that had entered a bad state.

The following services were affected during the incident:

Impact and response

The outage affected core traffic delivery across the network, with HTTP 5xx errors and elevated latency for CDN responses. Cloudflare has stated that any outage of its systems is unacceptable given its role in the Internet ecosystem, and that the incident is being used as the basis for process and system changes to prevent a recurrence.

Anatomy of the failure: a duplicate-row bug in the feature file

Every request routed through Cloudflare follows a fixed path: termination at the HTTP/TLS layer, then a core proxy system (code-named FL), and finally Pingora for cache lookups or origin fetches. During transit, domain-specific modules apply each customer’s configuration — WAF rules, DDoS protection, routing to the Developer Platform or R2 — against the traffic.

One of those modules, Bot Management, caused this incident. It includes a machine learning model that assigns a bot score to every request. The model reads a "feature" configuration file — a collection of traits (one per row) used to predict whether a request is automated. That file is refreshed every few minutes and published network-wide to keep pace with evolving bot tactics.

A change in the query behavior of our ClickHouse infrastructure (detailed below) caused the file generation logic to insert a large number of duplicate feature rows. The file, previously fixed in size, exceeded its limit. This triggered an error in the Bot Management module and caused the core proxy to return HTTP 5xx errors for any traffic depending on it. Workers KV and Access, which rely on the core proxy, were also affected.

Varying impact was observed depending on the proxy version. Customers on the new FL2 engine saw HTTP 5xx errors. Those on the legacy FL engine saw no errors, but bot scores were not generated correctly — all traffic received a score of zero, causing false positives for customers with bot-blocking rules. Customers not using bot scores in their rules saw no impact.

A concerning false lead

During the incident, our status page — hosted on infrastructure completely separate from Cloudflare — also went down. This was a coincidence, but it led some engineers to suspect a coordinated attack, especially in light of recent large-scale DDoS events. The status page displayed an error message to visitors:

The ClickHouse query change

The feature file generation logic relied on the behavior of distributed queries in our ClickHouse clusters. In such a cluster, data lives in shards. Distributed tables (powered by the Distributed table engine) live in a database called default and query underlying tables in a database r0, where data is actually stored per shard. Queries to distributed tables historically ran through a shared system account. Work was underway to make them run under the initial user accounts for better security and isolation.

Before the change, users querying ClickHouse metadata tables such as system.tables or system.columns would only see tables in the default database. Even though users had implicit access to r0, we made that access explicit at 11:05 so users could also see metadata for those underlying tables. This shift allowed distributed subqueries to run under the initial user, enabling fine-grained evaluation of query limits and access grants.

However, long-standing code assumed that metadata queries would only return columns for the default database, without filtering explicitly. After the change, a query like the following started returning "duplicates" — columns from both default and r0:

SELECT
  name,
  type
FROM system.columns
WHERE
  table = 'http_requests_features'
order by name;

The response effectively more than doubled in size, as shown in a simplified example, and this inflated row count was treated as a larger feature set for the Bot Management model.

Memory preallocation and the panic

Each module in the proxy service imposes limits to prevent unbounded memory consumption and as performance optimizations. The Bot Management system caps the number of runtime machine learning features at 200 — well above the ~60 currently used — precisely because memory is preallocated upfront. When the malformed file with more than 200 features propagated, the limit was hit and the system panicked. The failing check in the FL2 Rust code:

This resulted in the error trace: thread fl2_worker_thread panicked: called Result::unwrap() on an Err value, which translated to a 5xx response.

Wider impact and recovery

Workers KV and Cloudflare Access were both destabilized through their dependency on the core proxy. At 13:04, a patch allowed Workers KV to bypass the core proxy, reducing errors for all downstream systems that rely on it, including Access. The Cloudflare Dashboard also suffered because it uses Workers KV internally and Turnstile in its login flow.

Customers without an active dashboard session who relied on Turnstile were unable to log in during two windows: 11:30–13:10 and 14:40–15:30, per the graph below.

The first window was a direct consequence of Workers KV being affected; bypassing the core proxy at 13:10 restored that. The second window arose when the feature configuration data was restored: a backlog of retried login attempts overwhelmed the dashboard, causing elevated latency. Scaling control plane concurrency restored availability around 15:30.

Hardening for the future

Cloudflare engineers have begun work to prevent a recurrence:

• Treating internally generated configuration files as untrusted input, subject to the same validation as user-supplied data
• Expanding kill switches to disable individual features globally
• Eliminating the possibility of core dumps or error reports overwhelming system resources
• Auditing error-handling failure modes across all core proxy modules

This outage was the most severe since 2019, affecting the majority of core traffic. While past incidents have taken down the dashboard or newer features, today's failure interrupted the fundamental path through the network. The company acknowledged that such an event is unacceptable and has promised to build more resilient systems as a result of this analysis.