Why silent data corruption is the hard problem in AI reliability

Meta’s AI infrastructure spans thousands of hardware components connected across globally distributed data centers, with custom file systems and PyTorch applications supporting both training and inference. Hardware failures in components such as SRAMs, HBMs, processing grids, and network switch hardware significantly impact AI cluster reliability — over 66% of training interruptions in the Llama 3 herd resulted from such failures.

The challenges are compounded by accelerators that may be less reliable than CPUs due to complexity and limited telemetry, network complexity that can lead to misattributed failures, and errors in the GPU software stack requiring extensive configuration to correct. Meta classifies the hardware faults it observes into three broad categories.

Static errors

Static errors appear as binary states: a device either powers on or powers off. These are straightforward to identify at scale — simple health checks verify presence and configurations. As configuration and device scales grow in large training clusters, these faults occur more frequently but are easier to triage, root-cause, and repair, making them manageable.

Transient errors

Transient errors are categorized by their reproducibility and include load-dependent or partially observable faults, such as device issues from thermal runaway or random crashes from uncorrectable errors. Mitigation involves understanding manifestation conditions, with Meta’s scale aiding in triage and pattern matching. Advances in RAS telemetry have improved this process, and mitigation can also involve inducing conditions with artificial workloads in non-production stages to make faults more repeatable. Capturing transient states as “sticky” status values provides telemetry indications for hardware failures.

Silent errors

Silent errors, or silent data corruptions (SDCs), occur when hardware miscomputes without leaving detectable traces, causing applications to consume incorrect results. Often due to silicon defects, these errors can go unnoticed for long periods unless significant deviations are observed. They significantly impact large-scale services due to a lack of telemetry and continued consumption.

Historically, soft-error-related bitflips were reduced to one fault per million devices, but increased silicon density in accelerators means silent data corruptions now occur at about one fault per thousand devices — much higher than cosmic-ray-induced soft errors. SDCs are data-dependent, which creates an impractical exponential test space, and they also depend on device voltage, frequency, operating temperature, and life cycle. A device may fail computational checks only after months of use, indicating a state of “wear out,” so testing must be consistent, periodic, and frequent across the device’s life cycle.

Detection mechanisms Meta runs at fleet scale

To protect applications from SDCs, Meta employs three complementary detection mechanisms, detailed in the papers “Detecting Silent Errors in the Wild” and “Hardware Sentinel.”

  1. Fleetscanner: Captures performance outliers at scale with targeted micro-benchmarks for identifying hardware defects. Benchmark signatures are integrated into telemetry for non-benchmark-based detection. Tests run during maintenance operations such as firmware upgrades and hardware repairs, covering the entire fleet every 45 to 60 days. While it provides dedicated testing on hosts, it may be too slow for some SDCs.
  2. Ripple: Co-locates with workloads, executing tests in milliseconds to seconds for fleet-wide coverage in days. It overlaps test instructions across cores and threads, providing faster detection than Fleetscanner.
  3. Hardware Sentinel: A test-and-architecture-agnostic approach that evaluates application exceptions in kernel space. It identifies core-based anomalies as silent data corruption without requiring test allocations, operating solely in the analytical plane. Hardware Sentinel outperforms testing-based methods by 41% across architectures, applications, and data centers.

Combined, these three mechanisms provide one of the best in-fleet coverage at scale for detecting and protecting infrastructure against SDCs.

How SDCs manifest in AI workloads

AI applications such as training and inference have unique and more challenging implications for SDCs than general infrastructure.

Training: divergence with an illusion of progress

SDCs in training workloads lead to incorrect computations, affecting both forward and backward passes and causing divergence from the intended training path. While training is sometimes considered self-resilient to SDCs, that holds only for a limited subset of manifestations. SDCs persist across iterations, and the quantization of data values in AI training, which increases information per bit, exacerbates their impact and continuously increases divergence rates.

Two common cases of training divergence arise:

Not-a-Number (NaN) propagation

An SDC pushes a representable value into an incorrect representation, generating a NaN during training computations. The NaN propagates through subsequent computations, affecting the training iteration, accelerator domain, host domain, and eventually the entire cluster. This contagion can halt the cluster, and the source — often a few specific computations on a single accelerator — may be difficult to trace at cluster scale. Identifying and quarantining the offending accelerator and nodes is necessary to resolve the issue.

Corrupted gradient variance

An SDC affects gradient calculations, leading to gradient explosion, implosion, or local minima. The corruption stays within numeric bounds and is mistakenly treated as correct, affecting the entire cluster in synchronous training. Corrupted values are exchanged as true values, making training appear to progress without actual improvement. Over time, SDCs aggregate, causing major gradient divergences that can trap the algorithm in local minima or cause explosions or implosions.

This type of SDC is subtle and challenging to detect — effects may take weeks or months to observe. Unlike NaN propagation, these corruptions don’t trigger traps, making them harder to trace and rectify. They cause significant unproductive use of computational resources and training iterations, and until the offending device is identified, subsequent training remains risky.

Inference: corruptions bypassing policy boundaries

In inference applications, SDCs lead to incorrect results that, at operational scale, affect thousands of consumers. Persistent SDCs directly impact decisions made by recommendation engines or LLM outputs. These corruptions can bypass privacy or integrity policies, as they are not constrained by boundaries. Inference corruptions significantly reduce the efficacy of models trained with substantial computational resources, making seemingly benign inference use cases problematic at scale.

Operational impact

In training, visible faults halt the cluster, but SDCs create an illusion of progress that obscures the fault source. NaN propagation requires identifying the offending node; otherwise, restarts from checkpoints will eventually fail. Corrupted gradient variance prolongs this illusion until variances aggregate, making restarts ineffective. SDCs cause significant computational inefficiency, with a larger temporal impact than visible faults. In inference, triage requires costly telemetry at each substage, and clusters can’t be used until the offending node is identified — risking repeat corruption. Large deviations are easier to catch with anomaly detectors, but smaller ones require extensive debugging involving hundreds of engineers, halting production use cases and impacting reliable capacity.

Dealing with silent data corruptions in training fleets

Meta separates its SDC mitigation work into two buckets. Infrastructure strategies operate at the cluster level during triage, while stack strategies require coordination with the workload and make changes inside the software stack itself.

Infrastructure-level triage strategies

Reductive triage runs a binary search with small training iterations on progressively smaller clusters to isolate NaN propagation. The goal is to find a cluster small enough to reproduce the issue, quarantine the offending node, and reconstitute the cluster with fresh nodes from a saved checkpoint. The approach depends on SDCs being reproducible, which isn't guaranteed since they can depend on data, electrical, and temperature conditions. For corrupted gradient variance, a similar divide-and-triage method can work, though its effectiveness varies with training data and cluster size even under consistent hyperparameters.

Deterministic training runs a known-good model for a few iterations to confirm there are no NaNs or gradient divergences. This verifies computational failures that aren't data-dependent, since it guarantees correctness for a fixed set of values and training inputs.

Hyper-checkpointing creates checkpoints at increasingly higher frequencies to speed identification and isolation of the corrupting node. It protects training throughput while containing NaN propagation to a specific accelerator or host, which accelerates the triage and quarantine cycle.

Software stack strategies

Gradient clipping enforces value limits within the training workload. Computations that exceed the range are clipped, and NaNs can be caught at that point by setting them to a max or min value based on the operand sign. It works for some NaN representations but can introduce partial errors in certain cases.

Algorithmic fault tolerance bakes fault tolerance into the training algorithms themselves so they handle a range of data corruptions, reducing the need for detection and triage. It adds minimal overhead and has been demonstrated in CPU training, but requires understanding common defect modes and engineering investment across the stack.

Tri-variate computational training architecture uses shadow nodes in synchronous training. Training steps are repeated on different nodes at random iterations and progress is only confirmed after verification. If shadow and live nodes disagree, training halts and only those nodes are investigated while the rest continue with replacements. The scheme involves multiple shadow-node pools, a random training-node pool, and specified steps from the same checkpoint. It offers robust training at the cost of significant algorithmic changes and increased data movement and infrastructure overhead.

Parameter vulnerability factors identify which layers in machine-learning architectures are vulnerable and which are resilient, so vulnerable layers can map to protected hardware and resilient layers to unprotected hardware. This evaluation must scale with architecture evolution. Resilience costs area, power, or performance, so PVF enables targeted resilient design, particularly for inference.

Divergence detection keeps a distribution map per neuron to spot deviations from expected output distributions during inference. It's costly, so it's applied at selected sampling rates for large-scale workloads. Preserving per-neuron behavior for specific workloads makes it possible to detect corruptions during execution.

These methodologies offer different resilience levels with distinct operating points and engineering overhead. Choosing and orchestrating among them depends on the scale and intensity of the training or inference workload.

Performance faults and unknown unknowns

SDCs aren't the only fault mode at hyperscale. Meta's ServiceLab is a large-scale performance testing platform that catches tiny performance regressions, and Fleetscanner has flagged hundreds of performance outliers as an emergent fault mode alongside SDCs.

Existing mechanisms handle static, transient, and silent faults, but the full space of hardware fault variants is still not covered. The unknown unknowns demand agile solutions spanning the entire infrastructure and silicon lifecycle, and across the hardware-to-software and application stack, to reach first-class reliability operations.

Reliability timeline and industry collaboration

Meta's SDC work started with fleet issue identification in 2016, scaled detection in 2018, and shipped detection frameworks in 2019. By 2020, detection was integrated into accelerators, and Meta published "Silent Data Corruptions at Scale". In 2022, Meta released FleetScanner and Ripple and ran an academic RFP funding five winners.

In 2023, Meta worked with Google, Microsoft, ARM, AMD, NVIDIA, and Intel to improve server resilience and define test architectures and metrics. A joint RFP with the Open Compute Project selected six winners for cross-domain SDC research. By 2024, fleet-wide AI SDC detection methodologies were in production, with results shared through publications, tutorials, and talks at major conferences.

Applying fleet lessons to the MTIA family

Meta is applying these lessons to the Meta Training and Inference Accelerator (MTIA) family. The goal is to bring best-in-class, fleet-reliability practices into MTIA architecture and design, using a factory-to-fleet approach and revisiting reliability solutions continuously across the stack.

Factory-to-fleet lifecycle visibility

Finding unknowns early requires a complete factory-to-fleet view of the silicon lifecycle. Innovation is needed at every phase. In design and architecture, that means revisiting RAS solutions for scale, lifecycle debug hooks, and telemetry architectures that support tools like Hardware Sentinel, Fleetscanner, and Ripple. During validation and integration, novel yield analysis, manufacturing diagnostics, and fleet-signature-feedback-based detection can catch faults before shipping. For AI silicon fleets, user-space diagnostics with periodic testing, coverage maps, and control parameters help. Large-scale analytics such as Hardware Sentinel can detect early wear-out and data corruption. Robust firmware hooks and debug architecture feed findings back to design and architecture during fleet-scale issues.

Resilience beyond the silicon

Factory-to-fleet solutions handle silicon lifecycle resilience, but resilience must extend into firmware, compilers, kernels, and operating systems. That means investing in architectures for correctness-invariant instruction heterogeneity and enhanced telemetry for exception tracing. Granular firmware controls can improve telemetry once a fault is detected. At the software and application level, techniques like gradient clipping and algorithmic fault tolerance are essential for maintaining correctness amid corruptions. Experience with SDCs shows that in-line software resilience and test-agnostic analytical methods scale effectively for many SDCs with minimal investment, while testing-based approaches only cover specific instructions.

Hardware faults have a significant impact on AI training and inference in production. As cluster sizes and semiconductor complexity grow, fault complexity will increase exponentially. Solutions require factory-to-fleet coordination and stack-level resiliency, and for AI applications, reliability must be treated as a primary design consideration.