Keeping a global server fleet healthy
Facebook’s services depend on servers distributed across data centers worldwide. Hardware components fail for numerous reasons: mechanical wear in spinning disks, NAND flash pushed past endurance limits, corrosion from humidity, and manufacturing defects. Some level of failure is inevitable, so Facebook relies on systems like its cluster management system to limit service disruption. Four methodologies underpin this approach: automated detection and remediation, minimizing the performance cost of error reporting, using prediction to guide repairs, and automating root cause analysis.
Automated repair pipelines
Each server periodically runs MachineChecker, a tool that detects hardware and connectivity failures. When it raises an alert, Facebook Auto-Remediation (FBAR) picks it up and executes configurable remediation steps. Rate limits can be set to constrain how many servers undergo repair simultaneously, preserving capacity for running services.
If FBAR cannot restore a server, the case escalates to Cyborg, which handles lower-level actions such as firmware or kernel upgrades and reimaging. Failures requiring physical intervention generate tickets in a repair ticketing system for technicians.
More detail is available in the paper “Hardware remediation at scale.”
Reducing the performance cost of error reporting
MachineChecker detects problems by examining server logs. When hardware errors occur, the system typically flags them—for example, through a failed parity check—and sends an interrupt to the CPU for logging. These interrupts are high-priority; they halt normal CPU operation to process the error.
That halt hurts performance, even if only briefly. Logging correctable memory errors traditionally used a system management interrupt (SMI), which stalls all CPU cores. The newer correctable machine check interrupt (CMCI) stalls only one core, leaving the rest operational. While each stall typically lasts only a few hundred milliseconds, latency-sensitive services can be disrupted, and at fleet scale interrupts on a few machines can ripple into service-level degradation.
Facebook implemented a hybrid memory error reporting mechanism that combines CMCI and SMI. It avoids the performance cost of SMI while preserving accurate counts of correctable memory errors. The design is described in “Optimizing interrupt handling performance for memory failures in large scale data centers.”
Machine learning for repair recommendations
New hardware and software configurations constantly enter Facebook’s data centers, which means the auto-remediation rule set must keep pace. When automation fails, a manual repair ticket is created. But there is often a lag between introducing new components and incorporating new remediation rules. During that window, tickets can be classified as undiagnosed—no repair action suggested—or misdiagnosed—the suggested action proved ineffective. Technicians then diagnose manually, costing labor and downtime.
To close this gap, Facebook developed a machine learning framework that learns from past repair outcomes and predicts what actions would address current undiagnosed or misdiagnosed tickets. The system assigns a confidence threshold to each repair action based on the costs of incorrect predictions and the benefits of correct ones, then optimizes the order of actions. Because non-physical repairs such as reboots or firmware upgrades take less time, the algorithm often recommends attempting them first before scheduling component replacement.
This approach is detailed in the paper “Predicting remediations for hardware failures in large-scale datacenters.”
Automating root cause analysis across the fleet
Server logs record reboots, kernel panics, and out-of-memory events, plus software and tooling logs from production systems. At Facebook’s scale, examining millions of log entries—each with potentially hundreds of columns—jointly to find correlations is impractical.
Facebook implemented a scalable root-cause-analysis (RCA) tool that processes these entries to surface interpretable, actionable correlations. By pre-aggregating data with Scuba, a real-time in-memory database, they substantially improved the scalability of the FP-Growth pattern mining algorithm used in the framework. Additional filters on reported correlations sharpen interpretability. The analyzer is deployed broadly at Facebook for RCA on hardware component failure rates, unexpected server reboots, and software failures.
Further details are in the paper “Fast Dimensional Analysis for Root Cause Investigation in a Large-Scale Service Environment.”
