On August 30, 2020, CenturyLink/Level(3), one of the world’s largest Internet backbone providers, suffered a major outage that rippled across the globe. The incident affected not only CenturyLink’s own ISP customers but also countless third-party services, including some of Cloudflare’s clients. While the official postmortem from CenturyLink/Level(3) is still pending, an analysis of network telemetry and BGP data provides a clear picture of what happened, how traffic was rerouted, and why some services remained down for hours.

The First Signs of Trouble

At 10:03 UTC, Cloudflare’s monitoring systems detected a sudden spike in errors reaching customer origin servers. These manifest as “522 Errors,” which indicate a failure to establish a connection between Cloudflare’s network and the hosting infrastructure of its customers.

Cloudflare maintains direct interconnections with a wide array of network providers. When errors are detected on one path, automated systems immediately begin rerouting traffic through alternative carriers. This diversity is a core resilience feature: even when a major backbone fails, traffic can typically be shifted to other providers to keep services online.

In this instance, the automatic mitigation kicked in within moments of the error spike. Traffic that normally traversed CenturyLink/Level(3) was redistributed across other tier-1 networks, including Cogent, NTT, GTT, Telia, and Tata. By 10:09 UTC, the Network Operations Center had been alerted and was actively applying additional manual tweaks to address any gaps the automation had not covered. Within minutes, the system had disabled CenturyLink/Level(3) as a transit path in all 48 cities where Cloudflare interconnects with them.

The initial error spike dropped by half almost immediately after rerouting began, then fell to roughly 25 percent of peak levels as paths were optimized. Manual interventions shaved another 5 percent off the error count.

Why Some Customers Remained in the Dark

Despite these aggressive mitigations, a significant number of 522 errors persisted. The root cause of these residual failures lies in the topology of the Internet itself. CenturyLink/Level(3) is a massive network, and many hosting providers rely on it as their sole upstream connection — a single point of failure. When that link drops, there is simply no alternate path to reach those origin servers.

The problem was compounded by behavior observed in the BGP routing table. While CenturyLink/Level(3) was technically down, its routers continued to advertise routes that had been withdrawn. This effectively sent traffic into a black hole, as the network was unable to deliver packets but still claimed responsibility for them. For customers whose only connectivity was through CenturyLink/Level(3), or whose routes were still being advertised erroneously, there was no way to restore service until the underlying issue was fixed at the source.

On the consumer side, the outage took down a substantial portion of CenturyLink’s ISP service. This affected “eyeball” traffic — the users trying to reach websites and services. With CenturyLink unable to provide its users with an onramp to the broader Internet, those individuals were completely cut off. The global impact was measurable: a 3.5 percent drop in worldwide traffic during the incident window, almost entirely attributable to the loss of CenturyLink’s consumer network.

The BGP Storm

To understand the underlying cause, one must look at the Border Gateway Protocol (BGP), the system that routers use to announce which IP prefixes they can reach and to signal when those routes change or disappear. Under normal conditions, the global Internet sees roughly 1.5 to 2 MB of BGP updates every 15 minutes. At 10:04 UTC on August 30, that volume spiked to over 26 MB per 15-minute window.

This surge in updates points to severe instability within the CenturyLink/Level(3) backbone. Routes were being announced, withdrawn, and re-announced in rapid succession, indicating that routers were repeatedly crashing or losing connectivity with their peers. The spike began right at the onset of the incident and remained elevated for the entire duration, suggesting a feedback loop rather than a one-time failure event.

The Likely Culprit: Flowspec

CenturyLink/Level(3)’s own status update attributed the outage to a “bad Flowspec rule.” Flowspec is a BGP extension designed to distribute firewall rules across a network. It allows operators to push a rule to every router almost instantly, which is useful for mitigating attacks. However, a single incorrect rule can be catastrophic, as Cloudflare knows all too well — the company suffered its own Flowspec-induced outage over seven years ago and has since abandoned the protocol in favor of other mitigation techniques (though it remains widely used elsewhere).

Based on the evidence, the likely scenario involves an attempt to block malicious traffic using Flowspec. A rule meant to block an attack appears to have inadvertently instructed routers to drop BGP traffic itself. Once a router received this rule, it lost the ability to communicate BGP updates with its peers. This would explain the initial burst of withdrawals as routers went offline.

The continued elevation of BGP updates throughout the incident offers a possible explanation for the extended duration of the outage, however. If the offending Flowspec rule sat at the end of a long sequence of BGP messages, a repeating pattern could emerge: a router would reboot or recover, begin processing its queue of BGP updates, and work its way through legitimate rules until it hit the Flowspec rule that killed BGP again. The router would drop its BGP sessions, the rule would not be received, and the cycle would repeat. Each iteration would add more updates to the queue, placing increasing strain on router memory and CPU and potentially making the network harder to regain control of.

A Four-Hour Recovery Effort

The question on many minds is why it took over four hours to resolve. CenturyLink/Level(3) has a sophisticated NOC with immediate alerting, so the staff was certainly aware of the problem from the start. Several factors likely contributed to the delay, though — without their postmortem — these remain informed speculation.

A heavy load of BGP updates and the active Flowspec rule could have prevented engineers from logging into their own routers. Securing access to a router that is actively processing a massive queue of updates is not always straightforward. In some cases, other tier-1 providers appeared to de-peer with CenturyLink/Level(3) at their request — a cooperative measure to cut off the flood of BGP announcements, giving the network time to drain its queues and recover.

One scenario to consider is whether the offending rule originated from a CenturyLink/Level(3) customer. Many providers allow Flowspec peering, which gives downstream customers the power to issue their own rules. This is operationally useful but can make it considerably harder to trace the source of a harmful rule. Finally, the timing didn’t help: the incident began early on a Sunday morning, which can slow down response times even for first-rate operations. That said, an incident of this magnitude will certainly prompt a detailed retrospective.