Network Configuration Error Takes Cloudflare Down for 27 Minutes

Cloudflare experienced a 27-minute outage on July 17, 2020, caused by a configuration error in its backbone network rather than an attack or security breach. The incident, which ran from 21:12 to 21:39 UTC, cut traffic across the affected parts of the network by roughly 50 percent.

The failure originated from a routing change made while engineers were responding to an unrelated congestion issue between Newark and Chicago. A router in Atlanta was updated with a faulty configuration that directed all backbone traffic to that single location. The Atlanta router was quickly overwhelmed, and data centers connected to the backbone lost service.

Locations affected included San Jose, Dallas, Seattle, Los Angeles, Chicago, Washington DC, Richmond, Newark, Atlanta, London, Amsterdam, Frankfurt, Paris, Stockholm, Moscow, St. Petersburg, São Paulo, Curitiba, and Porto Alegre. Other locations continued operating normally throughout the incident.

Backbone Architecture

Cloudflare operates private lines between many of its data centers, forming a backbone used to carry traffic between locations without traversing the public Internet. This approach provides greater control over routing and can improve quality of service by avoiding Internet congestion points. For example, a request to an origin server in New York could be carried over the backbone to San Jose, Frankfurt, or São Paulo.

Incident Timeline

All timestamps are in UTC:

  • 20:25: Loss of backbone link between EWR and ORD
  • 20:25: Backbone between ATL and IAD is congesting
  • 21:12 to 21:39: ATL attracted traffic from across the backbone
  • 21:39 to 21:47: ATL dropped from the backbone, service restored
  • 21:47 to 22:10: Core congestion caused some logs to drop, edge continues operating
  • 22:10: Full recovery, including logs and metrics
Cloudflare outage on July 17, 2020 Embedded Image - QqRlsw

Following restoration, congestion at a core data center that processes logs and metrics caused some logs to be dropped. Edge network operations continued normally during this period, with full recovery of logging and metrics by 22:10.

Root Cause Analysis

The outage began when the network engineering team attempted to address backbone congestion in Atlanta by removing some of Atlanta's backbone traffic. Instead of deactivating the correct term in the router configuration, a one-line change caused all BGP routes to leak into the backbone.

{master}[edit]
atl01# show | compare 
[edit policy-options policy-statement 6-BBONE-OUT term 6-SITE-LOCAL from]
!       inactive: prefix-list 6-SITE-LOCAL { ... }

The relevant configuration term appears as follows:

from {
    prefix-list 6-SITE-LOCAL;
}
then {
    local-preference 200;
    community add SITE-LOCAL-ROUTE;
    community add ATL01;
    community add NORTH-AMERICA;
    accept;
}

This term normally sets the local-preference, adds communities, and accepts routes matching a specific prefix-list. Local-preference is a transitive property on iBGP sessions, meaning it propagates to the next BGP peer. The intended fix was to deactivate the term itself; instead, the prefix-list condition was removed.

With the prefix-list condition gone, the router was instructed to send all of its BGP routes to other backbone routers with an increased local-preference of 200. Local routes that edge routers received from compute nodes carried a local-preference of 100. Since higher local-preference values are preferred, traffic destined for local compute nodes was routed to Atlanta compute nodes instead. Atlanta thus began attracting traffic from across the entire backbone.

Remediation Steps

Cloudflare has implemented or scheduled two changes to prevent a recurrence:

  • A maximum-prefix limit will be introduced on backbone BGP sessions. This would have shut down the backbone in Atlanta during the incident; the network is designed to function without a backbone. Deployment was scheduled for Monday, July 20.
  • The BGP local-preference for local server routes has been changed, preventing a single location from attracting other locations' traffic in a similar manner. This change was deployed immediately following the incident.

Cloudflare noted this was the first outage experienced on its backbone, and while the team responded quickly to restore service, the company acknowledged the disruption caused to customers and Internet users during the incident period.