Risk-Driven Backbone Management in the COVID-19 Era
When the COVID-19 pandemic forced social interactions online, network operators faced an immediate, unplanned surge in traffic. For backbone networks spanning dozens of data center regions, this was compounded by the constant threat of fiber cuts, router misconfigurations, and power outages. To keep service levels stable through the stress test, Facebook ran a simulation system that predicts potential failures, quantifies their severity, and steers capacity deployment decisions accordingly.
This approach, detailed in a new study from Facebook, combines periodic risk simulation with traffic priority management and proactive capacity augmentation. It allowed the backbone to sustain high service availability and low latency during traffic spikes without major failures.
Modeling Failure and Measuring Risk
The strategy begins by defining risk metrics that reflect network service-level objectives (SLOs). Three categories are measured: demand loss, availability, and latency stretch. Each metric is computed across all possible failure scenarios in the network, which are enumerated by examining every component. The simulation estimates the likelihood of a scenario by using each component’s mean time between failures and mean time to repair. These statistics came from historical incident data, clustered and then fed through Bayesian regression models that incorporated features such as vendor, ownership, and geographical region.
The risk simulation runs periodically, consuming a fresh snapshot of the network topology and demand along with the failure scenarios to evaluate. Because the number of scenarios is large, each is sharded across multiple worker jobs. Those jobs execute the same traffic engineering logic as the SD-WAN controller to determine the routing response for each simulated failure. The aggregated results produce the risk metrics, which are logged and monitored continuously.
The risk metrics proved their worth at the onset of the pandemic. They reported a significant increase in demand loss—the highest traffic loss across any simulated failure scenario—along with reduced availability and higher latency across all quality of service (QoS) classes. Operators used these outputs to identify which failure scenarios would most harm network conditions in specific regions and then deployed capacity to those areas before failures materialized.
They also took a second approach: inspecting traffic flows from high-risk regions and classifying them by criticality. The network’s QoS classes are ordered by importance—infrastructure control (class 1), user traffic (class 2), internal applications (class 3), and bulk data transfer (class 4). A significant portion of latency-insensitive traffic was downgraded from class 3 to class 4. Lowering priority for such flows reduced the capacity needed to sustain the same SLO guarantees.
Why a Metric-Centric Approach Wins
Capacity additions to backbone networks typically require months or years of lead time, so operators normally procure capacity based on extrapolated traffic growth. COVID-19 broke that model, introducing a rapid ramp-up in usage that left little time for planning. The risk-driven method offered a response path: by leaning on simulated risk metrics, Facebook rerouted capacity enhancements toward the operational pain points identified by the system, quickly restoring the network to normal operating conditions.
The experience suggests that a metrics-centric backbone management framework can absorb rare external shocks as well as routine maintenance. For other operators, the lesson is that periodically computing risk metrics and using them to drive capacity decisions builds resilience into the network before an incident unfolds.
