PCIe faults at fleet scale
Millions of PCIe devices — accelerators, GPUs, NICs, SSDs — sit inside Meta's data centers, attached directly to motherboard slots or through PCIe switches. The diversity of vendors, firmware versions, and workloads makes fault monitoring and isolation a hard problem: failures span hardware, firmware, and software layers, and the same symptom can have different root causes across platforms.
Our approach is built around a set of detection and remediation tools. PCIcrawler is an open source, Python-based CLI that displays, filters, and exports PCI and PCIe bus/device information, including topology and Advanced Error Reporting (AER) errors, with both human-readable tree output and machine-parsable JSON. MachineChecker is an in-house tool that evaluates whether a server is production-worthy from a hardware standpoint; it runs as a CLI, a library, and a service. We also use an in-house hardware snapshot utility with modeling, plus a monitoring service that parses dmesg and System Event Logs (SELs) on millions of servers, recording PCIe-corrected error rates at intervals of 10 minutes, 30 minutes, one hour, six hours, and one day. Those rates are compared against per-platform, per-service thresholds.
For manual inspection and BMC-level work, we rely on IPMI Tool (for extracting SELs) and OpenBMC, a Linux distribution for baseboard management controllers. FBAR (Facebook Auto Remediation) executes automated responses to detected signals: it takes faulty servers out of production and files repair requests without human intervention. Finally, Scuba, our in-memory database, powers real-time analysis of all this telemetry.
Classifying what goes wrong
PCIe errors fall into categories with very different operational responses. Fatal uncorrected errors render the link unreliable and are always bad, even a single occurrence. Nonfatal uncorrected errors corrupt a specific transaction but leave the link functional. In practice, for any uncorrected error, we replace the hardware component — and sometimes the motherboard.
Corrected errors are trickier. They are recoverable by definition and usually cause no functional impact, but their rate matters. Above a certain frequency, they degrade performance enough to be unacceptable for specific applications. The challenge is that acceptable thresholds vary by platform and workload. We used our PCIe Error Logging Service to stream failure data into Scuba, then correlated events, system stalls, and error rates to determine per-platform thresholds. Above those thresholds, a hardware swap is the most effective fix.
Two error-reporting paradigms exist in PCIe: baseline capability (required of all components) and AER (an extended capability with more detailed reporting). We recommend that vendors adopt AER and PCIcrawler rather than shipping custom, vendor-specific tools, which are hard to parse and maintain, particularly across kernel versions and new hardware types. PCIcrawler leverages the PCIe AER driver to read this data uniformly.
Quiet problems: down-negotiated links and repeat offenders
Beyond errors, PCIe links can silently down-negotiate: running at half or quarter of expected speed, or at half, quarter, or even an eighth of the expected width. The hardware appears to work, so these faults are easy to miss without automated checks. In most such cases, reseating the component restores full link capability, so we try that before declaring hardware faulty.
A small fraction of faults clear on reboot, so we keep history of repair actions. For repeat offenders — the same component failing a predefined number of times within a window, even after reseats — we automatically mark it faulty and swap it. If the component swap does not resolve the problem, the motherboard gets replaced.
Repair trends are also monitored for anomalies. In one instance, Scuba visualizations of error timelines traced a down-negotiation problem to a specific vendor firmware release; working with the vendor, we rolled out updated firmware that fixed it. As a safety net, all remediations and repairs are rate-limited to prevent a bug in automation from mass-draining servers or unprovisioning, which could cause outages.
This methodology has extended hardware health coverage across the fleet. Each week, we detect, diagnose, remediate, and repair PCIe faults on hundreds of servers, and the system has fixed several thousand servers and components to date.
How PCIe faults are detected and resolved
Facebook’s PCIe fault-handling pipeline is a continuous loop of monitoring, analysis, remediation, and repair. It begins with MachineChecker, a service that runs periodically across the millions of servers and switches in production. Its checks cover PCIe link speed, link width, and accumulated corrected and uncorrected error rates.
For any flagged PCIe endpoint, the system identifies the parent device — called the upstream — using the PCIe topology data from PCIcrawler. Because a link is only as good as both of its ends, the system evaluates both the endpoint and the upstream device.
Speed and width expectations are derived from capability and configuration registers (LnkSta, LnkSta2, LnkCtl, and LnkCtl2). Expected speed is computed as the minimum of the upstream target speed and the capable speeds of both the upstream and the endpoint:
expected_speed = min (upstream_target_speed, endpoint_capable_speed, upstream_capable_speed)Current speed is taken from the actual negotiated speeds of both ends:
current_speed = min (endpoint_current_speed, upstream_current_speed)If current speed does not match expected speed, the link is suspect. Similarly, expected width is the minimum of the capable widths of the two connected devices. When the upstream device is running at a width above the expected value, the link is flagged as bad.
Error logging and thresholding
Independently of MachineChecker, the PCIe Error Logging Service records corrected and uncorrectable error rates on every hardware server in a structured JSON format. MachineChecker consumes these logs and applies platform-specific thresholds stored in Configerator, Facebook’s configuration management system. Even a single uncorrected error event is enough to declare a server faulty. Corrected error rates are judged against preset, per-platform limits, which operators can adjust without rolling out new code.
PCIcrawler, also preinstalled on all servers, provides additional visibility into PCIe AER (Advanced Error Reporting) issues. The fault data, spanning link width, link speed, and AER information, is cross-referenced with an in-house hardware inventory service that maps each PCIe address to a physical component.
From alert to repair
When a fault is confirmed, MachineChecker generates an alert or alarm. The alert includes actionable details for data center operators — the component’s location, vendor, and model information — taken from the hardware mapping service. This reduces the time needed to locate and swap a failing part.
Application production engineers can subscribe to these alerts and build custom workflows for monitoring, remediation, or repair. In many cases, an automated remediation runs first. The process can be tuned per fault type; for a well-understood issue, remediation might be restricted to a firmware upgrade. If the remediation attempts do not succeed, a hardware repair ticket is automatically filed, prompting operators to replace the faulty component or entire server with verified-good hardware.
Rate limiting is placed at several stages in the pipeline. This is a deliberate safety measure: it prevents bugs in the detection code or a runaway remediation workflow from draining or unprovisioning machines at scale, which would create larger service disruptions than the original hardware fault.
This workflow has already identified and fixed thousands of servers and components and is currently handling hundreds of PCIe-related repairs per week. By feeding these results back into how the fleet is monitored and repaired, Facebook has improved the reliability and longevity of its hardware.
