Outage at 19 Cloudflare locations takes down a major slice of global traffic
Cloudflare suffered an outage on June 21, 2022, affecting 19 of its data centers. These locations handle a significant share of the company's global traffic, and the incident began at 06:27 UTC. The first affected data center was restored at 06:58 UTC, and all were back online by 07:42 UTC. Services in other parts of the world remained operational. Cloudflare confirmed the outage was not caused by an attack but by an internal error.
What changed: A look at the Multi-Colo PoP architecture
Over the last 18 months, Cloudflare has been migrating its busiest locations to a new internal architecture called Multi-Colo PoP (MCP). The 19 locations involved in the incident—including Amsterdam, Ashburn, Chicago, Frankfurt, London, Los Angeles, Mumbai, Osaka, São Paulo, Singapore, Sydney, and Tokyo—had already been converted.
The MCP design is based on a Clos network topology, which adds a routing layer made up of a mesh of connections. This layer, represented by the spines in the diagram below, is intended to make it easier to disable or enable parts of the internal network for maintenance or troubleshooting.
This architecture has delivered significant reliability improvements and enabled maintenance without customer-facing disruption. But because these 19 locations carry such a large share of traffic, a failure in this routing layer has outsized consequences.
How the incident escalated
The root cause was a change to BGP prefix advertisement policies. BGP policies determine which IP prefixes a network advertises to or accepts from its peers. A policy change can cause a previously advertised prefix to be withdrawn, making the corresponding IP addresses unreachable on the Internet.
Cloudflare was rolling out a change to standardize BGP communities attached to a subset of prefixes it advertises, specifically adding informational communities to site-local prefixes. These site-local prefixes enable communication between servers in a location and connections to customer origin servers.
A re-ordering of terms in the policy caused a critical subset of prefixes to be withdrawn. The sequence of events was:
- 03:56 UTC: Change deployed to first location—older architecture, no impact.
- 06:17: Change deployed to busiest non-MCP locations, no impact.
- 06:27: Change reached MCP-enabled locations, deployed to the spines. Incident begins.
- 06:32: Internal incident declared.
- 06:51: First router change made to verify root cause.
- 06:58: Root cause identified; reverts begin.
- 07:42: All reverts completed. This was delayed because network engineers' changes crossed each other, sporadically re-introducing the problem.
- 08:00: Incident closed.
The affected data centers make up only 4% of Cloudflare's network, but the outage impacted roughly 50% of total successful HTTP requests and a similar share of egress bandwidth.
What actually broke in the BGP policy
The change that was deployed to the MCP spines had what appeared to be a harmless modification to BGP policy—adding informational communities. However, the diff on the spines contained a subtle but critical difference. In the policy configuration, the terms for advertising site-local prefixes (4-ADV-SITE-LOCALS and 6-ADV-SITE-LOCALS) were moved to the bottom of the term list. This placed them after the REJECT-THE-REST term, an explicit reject-all rule. As a result, site-local prefixes were no longer advertised at all, removing direct access to all impacted locations and breaking connectivity to origin servers.
term REJECT-THE-REST {
then reject;
} Beyond losing origin access, the withdrawal of site-local prefixes also broke Cloudflare's internal load balancer, Multimog, which relies on these routes to forward requests between servers within an MCP. Without functioning load balancing, smaller compute clusters in an MCP received the same amount of traffic as the largest clusters, causing them to be overwhelmed.
The human and process failures
The rollout procedure was not the first at fault here. Cloudflare created a Change Request ticket, which included a dry-run and a stepped rollout procedure, and the change was peer reviewed by multiple engineers. The problem was that the steps were not granular enough to catch the error before it hit all of the MCP spines at once.
Although the company used a stagger procedure for this change, that procedure did not include an MCP data center until the final step. This procedural gap meant the MCP architecture, which was designed to improve availability, was never tested with the new change in a smaller, representative environment before a full rollout.
Remediation steps: What Cloudflare is fixing now
Cloudflare has outlined immediate follow-up work across three areas:
- Process: Introduce MCP-specific test and deployment procedures for all changes, ensuring changes are validated in an MCP context before hitting all MCP locations. Include MCP data centers explicitly in stagger policies.
- Architecture: Redesign the BGP policy statement that caused the incorrect routing advertisement so that an unintentional re-ordering of terms cannot cause a similar issue.
- Automation: Build automation that enforces a robust stagger policy for network configuration rollouts and provides an automated “commit-confirm” rollback that can quickly undo a bad change. The improved stagger would have significantly reduced the blast radius, and automated rollback would have drastically shortened the recovery time.
The combination of a poorly ordered rollout and a lack of automated rollback turned a single configuration change into a 75-minute, multi-region outage. Cloudflare has stated it will continue to look for other gaps that could cause a recurrence.
