Detecting silent errors in the wild: Combining two novel approaches to quickly detect silent data corruptions at scale

Silent data corruption at hyperscale: two detection strategies that complement each other

Silent data corruptions (SDCs) are corrupted computations that produce no error flag, log entry, or other notification. A machine carrying an SDC fault can keep serving incorrect results that ripple outward into dependent services and eventually surface as application-level data loss or logic errors. The root causes live in silicon: datapath defects, thermal variance, and component aging. Because the failures are silent, they are extremely hard to catch once a server is running production traffic.

Manufacturers and integrators typically test CPUs for only a few hours or a few days before deployment, with sampling-based verification after that. That kind of coverage does not translate to a fleet once hardware is in service. At Meta, engineering teams have spent multiple years running two complementary detection mechanisms — out-of-production "opportunistic" testing and in-production "ripple" testing — and measuring what each one catches. The key finding is that neither approach alone is sufficient; together they close most of the detection gap.

The testing funnel: what each method costs

Testing and production workloads contend for the same machines. Out-of-production tests carry a ramp-up and ramp-down cost each time the machine transitions from tests back to service. In-production tests avoid that transition but can leave residual configurations behind that degrade workload performance. Scaling either approach fleetwide magnifies the overhead and the operational complexity.

Opportunistic testing: piggybacking on maintenance windows

Large fleets constantly cycle machines through maintenance events: reboots, kernel upgrades, firmware updates, reimages, provisioning, and repairs. Opportunistic testing hooks into those events to run longer, more intrusive SDC tests while the machine is not serving traffic. Meta's internal facilitator tool for this mode is called Fleetscanner.

Because the tests run idle, they can be far more aggressive — test runtimes in the order of minutes are acceptable. The trade-off is that a fault can emerge, go undetected, and keep corrupting data in the interval between two opportunistic windows. Averaged across the fleet and accounting for machine type, services see an opportunistic test roughly once every 180 days. The team has added fine-grained controls so that more sensitive service tiers can trigger shorter cadences.

The downside is measurable: scheduled maintenance takes longer because SDC tests are appended to every opportunistic slot. Test time and frequency are therefore continuously balanced against detection value. In practice, opportunistic testing has run more than 68 million tests at the fleet level, totalling approximately 4 billion seconds of test time.

Ripple testing: detection inside production traffic

Opportunistic windows were not enough. Failures still surfaced between test intervals, depended on data patterns that identical test vectors never matched, or appeared only after shifting between operating modes. Ripple testing attacks all three gaps by running SDC checks in colocation with production workloads.

The mechanism works by shadowing workloads and periodically injecting bit patterns with known expected results into the instruction stream, on a cadence defined per service. This gives unbroken fleetwide coverage without pulling machines out of service. Granular controls tune test interval, duration, and device selection per workload.

The runtimes are dramatically shorter. Where opportunistic tests run minutes, ripple tests run in the hundreds of milliseconds — roughly 1000x less. Because ripple testing is scale-driven and uses unique seeds on each run, it also increases the number of transition events a CPU sees, catching failures that only materialize after thousands of iterations with different data. The team has shared early results with CPU vendors, who have incorporated similar techniques into their own tooling.

The colocation footprint is reportedly negligible compared with routine management activity. The framework has been in production for about two years and processes roughly 2.5 billion unique test seeds per month, for a fleetwide total of nearly 100 million seconds of in-production test time.

Why you need both

The two methods specialize in different parts of the fault spectrum. In-production testing is fast: it catches approximately 70 percent of fleetwide data corruptions within 15 days. The remaining fault population breaks down differently. Opportunistic testing is needed to find another 23 percent of uniquely faulty CPUs, a process that takes up to six months. The residual 7 percent only surfaces through repeated ripple events spread across the fleet.

That pattern — nearly instant broad coverage from ripple, deep long-tail coverage from opportunistic — is why the team insists on running both models at scale. The same analysis is also revisited as test vectors, fault categories, and hardware change; with a different defect profile the coverage split could shift. Neither the distribution of silent faults nor the ideal detection mix should be assumed static.

Before these systems were in place, hardware got only burn-in level screening at the factory and a few hours of post-integration testing. Services exposed to SDCs can run for months with corruption brewing underneath, which then takes many more months to debug once the software-level residue surfaces. That cascading damage window is the strongest argument for accelerated detection everywhere in the fleet — and for treating SDC detection as an active infrastructure priority rather than a one-time screening step.