What Took Facebook Offline on October 4
Now that services have returned to normal, we're sharing more detail about what caused the global outage on October 4 and what our engineers encountered while recovering the network.
The outage originated in the system that manages our global backbone capacity. Our backbone is the private fiber-optic network connecting our data centers, spanning tens of thousands of miles. Some of those data centers are large facilities housing millions of machines that store data and run heavy computational loads. Others are smaller points of presence that connect our backbone to the broader internet and to users.
When an app fetches your feed or messages, the request travels from your device to the nearest small facility, then crosses the backbone to a larger data center where data is processed and sent back the same way. Routers managing this traffic decide where all that data should go. Routine maintenance on this infrastructure frequently requires taking part of the backbone offline, such as to repair fiber or update router software.
During one such maintenance job on October 4, a command meant to assess global backbone capacity inadvertently shut down all backbone connections, severing links between Facebook data centers and the wider internet. Our systems are designed to audit such commands to prevent mistakes, but a bug in that audit tool let the faulty command through.
A second issue amplified the impact. One function of those smaller facilities is to respond to DNS queries, translating simple hostnames into IP addresses. These authoritative name servers advertise themselves to the rest of the internet using the border gateway protocol (BGP). To avoid routing traffic to an unhealthy link, those DNS servers disable their BGP advertisements if they cannot reach our data centers. With the backbone fully down, the locations declared themselves unhealthy, withdrew their BGP routes, and became unreachable—even though they were still operational. The rest of the internet had no way to find our servers.
Recovery Was Slowed by Design
The failure itself was instantaneous. But the response faced two obstacles. First, our engineers could not access the data centers through normal remote means, because their networks were down. Second, the DNS loss broke many of the internal tools that our engineers rely on for investigating outages.
With primary and out-of-band network access unavailable, engineers were dispatched to the data center sites. Reaching the systems was slow: those facilities have strict physical and system security, and their hardware and routers are deliberately hard to modify even with physical access. Activating the secure access protocols needed to get people inside and working took additional time.
Once backbone connectivity was restored across our data center regions, services resumed. But bringing everything back up at once posed its own risk. During the outage, individual data centers recorded power-consumption dips of tens of megawatts. A sudden ramping back up of that load can endanger electrical equipment and caches.
For that phase of the recovery we drew on "storm" drills that simulate major failures by taking a service, data center, or entire region offline and stress-testing the systems that bring it back. The experience from those drills guided us in re-enabling services gradually. Services recovered quickly without further systemwide failures. This was the first incident where we'd simulated losing the entire global backbone, however, and we will definitely look for ways to simulate events like this going forward.
Our post-incident review has already begun. One notable observation is that the security hardening we have done to prevent unauthorized access also slowed our recovery when the failure was caused by our own error. That tradeoff is worth keeping: increasing day-to-day security at the cost of a slower recovery from what we hope is a rare event. Moving forward, our challenge is to strengthen testing, drills, and fault tolerance so such an event happens as infrequently as possible.
