Route leak post-mortem: a 32-second BGP mistake and the bias that hid it

On February 1, 2022, a configuration error during a scheduled maintenance window caused a route leak of up to 2,000 Internet prefixes to one of Cloudflare's transit providers. The leak lasted 32 seconds, with a second leak of 7 seconds occurring later. No traffic anomalies or customer impact were observed on Cloudflare's network, but external parties may have been affected.

This is a case study in how even mature networks with peer-reviewed deployment processes can fail, and how confirmation bias can delay detection of the real problem.

Maintenance gone wrong

The incident began as a routine "hot-cut" migration: Cloudflare was upgrading an existing Internet transit link in Newark to a higher-capacity connection. Both Cloudflare and the ISP would change cabling and router configurations simultaneously during a conference call to minimize downtime.

The migration started at 10:45 UTC, off-peak. Key events unfolded as follows:

  • 11:17 — New fiber link connected, BGP sessions established with export/import filters in place
  • 11:22 — Link deemed ready for production; filters removed
  • 11:23 — ISP router tears down BGP sessions due to maximum prefix-limit violation
  • 12:08 — ISP raises prefix-limits; sessions briefly reestablished, then shut down again
  • 12:14 — Root cause identified and configuration corrected

Before deeming the link production-ready, the engineer checked optical light levels and interface errors. Both were satisfactory. The decision to remove the BGP filters followed a peer-reviewed deployment plan. But the plan contained an oversight: no export policy was configured to restrict BGP announcements to only Cloudflare's and its customers' prefixes.

Upon removing the filters, the router began advertising all known prefixes to the ISP. The ISP's maximum prefix-limit — configured just 40 prefixes below the number Cloudflare was advertising at that site — shut the session down immediately, preventing broader impact.

The bias that delayed the fix

Here's where things took an unexpected turn. When the BGP session was torn down, the network engineering team didn't investigate. The prefix-limit value seemed too low for Cloudflare's network size, so the session failure appeared entirely explainable. The team asked the ISP to raise the limit, which was done with approval on their side.

When the session was reestablished at 12:08 and immediately shut down again — the new limit was also exceeded — the engineer finally questioned whether something else was wrong and found the missing export policy.

The delay highlights confirmation bias: the operator saw the failure as expected behavior and had no reason to look deeper. It was only when the second shutdown occurred, with a limit that had been raised, that the true cause became apparent.

Mitigation and defense in depth

Cloudflare has since deployed an implicit reject policy for BGP sessions: if no import/export policy is configured for a BGP neighbor or neighbor group, the session will reject all prefixes. This change is now in production.

Route leaks remain a systemic problem across the Internet, and multiple technical controls exist to mitigate them:

  • BGP policies and filters restrict exports to only the network's own prefixes and those of its customers.
  • RPKI (Resource Public Key Infrastructure) helps validate that received prefixes are authorized by the holding entity.
  • Maximum prefix-limits act as a last line of defense, tearing down sessions that exceed expected prefix counts.

A 2017 Internet standard addresses precisely the failure mode Cloudflare experienced: RFC8212, titled "Default External BGP (EBGP) Route Propagation Behavior without Policies," updates the BGP-4 standard (RFC4271) to specify that EBGP sessions should reject routes by default unless explicit policies exist.

Vendor support varies. On Junos, the behavior is available starting with Junos OS Release 20.3R1 via:

defaults ebgp no-policy reject-always under the protocols bgp hierarchy. For older Junos versions, an equivalent can be configured by applying a REJECT-ALL policy as import/export on the BGP protocol level, with the caveat that this also affects iBGP sessions, unlike the native RFC8212 implementation. The RFC8212 repository on GitHub tracks implementation status across various network operating systems.

A human error with systemic fixes

While the configuration error stemmed from human error, it could have been detected and contained much faster without confirmation bias. Cloudflare says it will increase efforts to derive device configurations from an intended state through network automation, and will continue training staff to recognize cognitive biases that can mask production issues.

The incident also reinforces the case for RPKI deployment: route leaks can happen even to experienced network operators, and technical safeguards are the most reliable means of reducing their impact — whether the leak is accidental or malicious.