A perfect storm: two separate failures, one long outage

On June 20, 2024, Cloudflare suffered a 114-minute incident that degraded performance and availability for many Internet properties. At its peak, between 1.4% and 2.1% of all HTTP requests to Cloudflare's CDN returned a generic error page, and the 99th percentile Time To First Byte (TTFB) latency nearly tripled. The outage was the result of two distinct and unrelated events that overlapped in time.

The first event began at 17:33 UTC when automated network monitoring detected performance degradation and re-routed traffic in a way that caused congestion on the backbone between data centers. The second, and more impactful, event was caused by a latent bug in the rate-limiting system, exposed by a newly deployed DDoS mitigation rule. Cloudflare states that there is no evidence the triggering traffic was intentionally exploiting this bug, nor any indication of a security breach.

Root cause: a poisoned process

Cloudflare routinely applies rate limits to HTTP requests. These limits can be configured by customers or activated automatically when DDoS rules detect suspicious activity. Typically, rate limits are enforced based on a visitor's IP address. However, since many users and devices often sit behind a single IP due to carrier-grade NAT, IP-based limiting is broad and can catch legitimate traffic.

As part of a routine update, Cloudflare developed a new DDoS rule to prevent a specific type of observed abuse. The rule itself functioned correctly, but in a particular suspect traffic case, it exposed a latent bug in the pre-existing rate-limiting component. This bug caused the process handling a specific form of HTTP request to enter an infinite loop, effectively "poisoning" it.

Network balancing basics

To understand how a few poisoned processes snowballed into a global incident, it helps to understand Cloudflare's traffic management architecture. Every packet entering the anycast network passes through Unimog, the edge load balancer. Unimog routes each packet to an appropriate server, often aiming to keep CPU load uniform across all servers within a data center—even if that means forwarding traffic to a different physical location.

For a broader, network-wide view, Cloudflare relies on Traffic Manager. It ingests signals such as CPU utilization, HTTP request latency, and bandwidth usage to make rebalancing decisions across data centers. Traffic Manager includes safety limits to prevent outsized traffic shifts and models the expected load on destination locations before acting.

Timeline of the incident

All times are UTC on 2024-06-20:

  • 14:14: Gradual deployment of the new DDoS rule begins
  • 17:06: DDoS rule is deployed globally
  • 17:47: First HTTP request handling process is poisoned
  • 18:04: Incident declared automatically due to high CPU load
  • 18:34: Service restart appears to recover a server; full restart tested in one data center
  • 18:44: CPU load normalizes in the tested data center after restart
  • 18:51: Continual global reloads of servers with stuck processes begin
  • 19:05: Global error rate peaks at 2.1% service unavailable / 3.45% total
  • 19:05: Traffic Manager begins recovery actions
  • 19:11: Error rate halves to 1% service unavailable / 1.96% total
  • 19:27: Error rate returns to baseline levels
  • 19:29: DDoS rule identified as likely cause of poisoning
  • 19:34: DDoS rule fully disabled
  • 19:43: Engineers stop routine restarts of affected services
  • 20:16: Incident response stood down

Measuring the impact

The first metric below shows the percentage of eyeball (inbound external) HTTP requests that received an error because a poisoned service could not be reached. There was an initial spike to 0.5%, followed by a larger increase to 2.1% before service reloads began to take effect.

BLOG-2459 Embedded Image - vcudXg

A broader view of all 5xx responses returned to eyeballs—including those from origin servers—peaked at 3.45%. The gradual recovery between 19:25 and 20:00 UTC is visible as Traffic Manager completed its re-routing. The dip at 19:25 UTC coincides with the last large reload; the subsequent error uptick was primarily upstream DNS timeouts and connection limits, consistent with high and unbalanced load.

BLOG-2459 Embedded Image - jZXVSm

Latency measurements at the 50th, 90th, and 99th percentiles showed an almost 3x increase in TTFB at p99 during the window.

BLOG-2459 Embedded Image - LLHam4

Remediation and prevention

Cloudflare had already been working on two related improvements prior to the incident: expanding backbone capacity in the affected data centers, and enhancing network mitigations to account for available capacity on alternative paths when making rebalancing decisions. Beyond those efforts, the company has also made changes to help prevent the recurrence of the rate-limiting bug exposed by the new DDoS rule.

Cloudflare apologized for the impact and confirmed that both the DDoS rule and the rate-limiting component have been adjusted. The backbone congestion event lasted from 17:33 to 17:50 UTC, and the rate-limiting issue from 17:47 to 19:27 UTC; the overlap of these windows compounded the overall disruption.

The mechanics of the bug

BLOG-2459 Embedded Image - 9j2Fm4
Global percentage of HTTP Request handling processes that were using excessive CPU during the event Starting at 14:14 UTC on June 20, Cloudflare began phasing in a new DDoS mitigation approach across its network. The system combines rate limiting with browser cookies so that legitimate clients falsely flagged as attackers can still get through. When the new logic deems a request suspicious, it runs through a fixed sequence:
  1. Check for a valid cookie; if none exists, block the request.
  2. If a valid cookie exists, register a rate-limit rule keyed on the cookie value for later evaluation.
  3. After all other active DDoS mitigations have executed, apply the rate-limit rules.
The asynchronous workflow exists because blocking a request without a rate-limit rule is cheaper, giving other rule types a chance to run first. In simplified form: 
for (rule in active_mitigations) {
   // ... (ignore other rule types)
   if (rule.match_current_request()) {
       if (!has_valid_cookie()) {
           // no cookie: serve error page
           return serve_error_page();
       } else {
           // add a rate-limit rule to be evaluated later
           add_rate_limit_rule(rule);
       }
   }
}

evaluate_rate_limit_rules();
To evaluate rate-limit rules, the system derives a key per client to locate the corresponding counter, comparing it against the target rate. The key is normally the client's IP address, but alternatives such as a cookie value are supported. Cloudflare reused existing rate-limit machinery to build the cookie-based key. In pseudocode: 
function get_cookie_key() {
   // Validate that the cookie is valid before taking its value.
   // Here the cookie has been checked before already, but this code is
   // also used for "standalone" rate-limit rules.
   if (!has_valid_cookie_broken()) { // more on the "broken" part later
       return cookie_value;
   } else {
       return parent_key_generator();
   }
}
Two defects in this key-generation function, when combined with a particular pattern of client requests, produced an infinite loop inside the HTTP request handler:
  1. The dynamically generated DDoS rate-limit rules invoke internal APIs in ways never anticipated. As a result, the parent_key_generator ended up pointing at get_cookie_key itself—so when that branch was reached, the function called itself without bound.
  2. The rule is only installed after cookie validation succeeds, implying a second validation would agree. But the has_valid_cookie_broken helper used in the key path is not identical to the original check. If a client sends several cookies and some are valid while others are not, the two functions can disagree.
Those two flaws compound into the loop: the broken validation incorrectly reports an invalid cookie, so get_cookie_key takes the else branch and invokes itself again. Most languages guard against runaway recursion by capping stack depth; a call made once the limit is reached throws a runtime error. A first reading suggests that was happening here, with requests dying on a stack repeating the same calls. That was not the case. The logic is written in Lua, which implements a proper tail call optimization. When a function's final action is another function call and no local state will be needed afterward, the current stack frame can be replaced rather than extended. Thus the recursion became a loop that never grows the stack—it just burns one full CPU core and never returns. Any single request matching the rule and carrying mixed valid/invalid cookies permanently poisoned the process handling it, rendering it unable to service further requests.

Cascade to global impact

Each server runs dozens of these processes, so one poisoned process alone is inconsequential. But the failure propagated through Cloudflare's traffic management layers:
  1. Rising CPU use on a server caused Unimog to divert new traffic to servers with healthier processes and lower utilization.
  2. As the data center's overall CPU climbed, Traffic Manager shifted traffic to other data centers. The rerouting did nothing to fix poisoned processes, so utilization stayed high and Traffic Manager kept pushing traffic away.
  3. Both layers redirected traffic that included the poisoning requests themselves, dooming the servers and data centers that absorbed the redirected load in the same way.
Within minutes, multiple data centers were saturated with poisoned processes. Traffic Manager had rerouted as much as its automation safety limits allowed, and suitable destinations with spare capacity were becoming scarce. The first poisoned process appeared at 17:47 UTC. By 18:09 UTC—five minutes after the incident was declared—Traffic Manager was shifting substantial volumes of traffic out of Europe:
BLOG-2459 Embedded Image - xPntcl
A summary map of Traffic Manager capacity actions as of 18:09 UTC. Each circle represents a data center that traffic is being re-routed towards or away from. The color of the circle indicates the CPU load of that data center. The orange ribbons between them show how much traffic is re-routed, and where from/to. The regional CPU data explains why. Western Europe had already lost 10% of its HTTP request handling capacity and Eastern Europe 4%, during that region's peak traffic period:
BLOG-2459 Embedded Image - SREcBa
Percentage of all the HTTP request handling processes saturating their CPU, by geographic region With many servers partially disabled, the surviving processes in several data centers could not keep pace, and Cloudflare returned minimal HTTP error responses to users.

Detection and response

Automated alerts fired at 18:04 UTC once sustained global CPU usage crossed a threshold. However, the incident responders on duty were simultaneously working an open network congestion incident triggered by backbone issues. Early investigation focused on whether the CPU spike correlated with that congestion. The evidence pointed elsewhere: locations with the highest CPU were the ones receiving the least traffic, which is not the signature of a network-originated problem. That shifted the effort onto two tracks:
  1. Determining whether restarting poisoned processes allowed them to recover, and if so, mass-restarting the service on affected servers
  2. Isolating what triggered the CPU-saturation state
Twenty-five minutes after the incident was declared, a test confirmed that restarts recovered a sample server. Five minutes after that, Cloudflare broadened the restarts—at first data-center-wide, then targeted at servers carrying the highest number of poisoned processes. One team conducted rolling restarts while another kept working to identify the root trigger. At 19:36 UTC the new DDoS rule was disabled globally. After one final round of mass restarts and monitoring, the incident was declared resolved. The event also exposed a latent bug in Traffic Manager. Faced with the unusual conditions, it could crash and recover via a graceful restart that put its activity on hold. The bug first triggered at 18:17 UTC and recurred repeatedly between 18:35 and 18:57 UTC. During two windows in that span—18:35-18:52 UTC and 18:56-19:05 UTC—Traffic Manager issued no routing actions at all. Consequently, even after services recovered in the hardest-hit data centers, almost all traffic still bypassed them. Monitoring alerted engineers at 18:34 UTC, and the Traffic team had a fix written, tested, and deployed by 19:05 UTC. Service restoration immediately improved once routing actions resumed.

Fixes and forward-looking changes

Cloudflare recovered by repeatedly restarting the poisoned service until engineers identified and rolled back the rule that caused it. The new DDoS rule remains disabled and will not be reactivated until the broken cookie validation check is fixed and Cloudflare is confident the situation cannot recur. The company has outlined follow-up measures across design and process:
  • Design: The rate-limiting code used by the DDoS module is legacy. Customer-configured rate limit rules rely on a newer engine with more current safeguards. The gap between those two implementations is a key contributor to this failure.
  • Design: Cloudflare is evaluating ways to prevent unbounded tail-call loops inside the affected service. Longer term, a replacement service is in early implementation, designed to cap both the uninterrupted and total execution time per request.
  • Process: The new rule was initially rolled out to a small number of production data centers for validation, then broadened to all data centers hours later. Staging and rollout procedures will be strengthened to shrink the potential blast radius of future changes.
Two back-to-back incidents affected a wide swath of CDN and network customers on June 20. The first, backbone congestion, was resolved through automated remediation. The second was handled by restarting the faulty service repeatedly while the triggering DDoS rule was identified and deactivated. Cloudflare stated that the conditions required to activate the latent bug no longer exist in its production environment, with additional fixes and monitoring being deployed.