Debugging silent corruption that skips decompressed files
Silent data corruption—errors that escape detection by the larger system—is a known hazard in large-scale infrastructure. It can propagate upward through the stack, surface as application-level failures, cause data loss, and take months to isolate. This work, based on a paper from engineers running fleets of hundreds of thousands of machines, documents a real-world case of corruption in a data center CPU and lays out best practices for detecting and remediating such faults.
The investigation started with an ordinary operation: in large-scale infrastructure, files are compressed while idle and decompressed on read. Millions of these operations run daily. A bug manifested in the decompression pipeline—a file with a non-zero size was fed to the decompression algorithm, but the file size computation returned zero. As a result, the file was skipped and never written to the decompressed output database, leaving downstream databases with missing files and applications unable to recover certain compressed entries. The failure was intermittent and workload-dependent, making it exponentially harder to diagnose.
From fleet-wide symptom to single-core root cause
Reproducing the issue at scale proved complex. Engineers reduced a multi-machine querying failure down to a single-machine workload, where the defect remained sporadic. Narrowing further, they found the failure was tied to multi-threading: single-threading the workload eliminated the timing-based randomness, but the fault became consistent for a subset of data values on one specific core of the machine.
Repeated iterations eventually isolated the trigger. The computation
Int (1.153) = 0
as input to the math.pow function in Scala always produced a result of 0 on Core 59 of the defective CPU. Altering the exponent—for instance, Int (1.152) = 142—yielded accurate results. The issue was intrinsic to the core and the data values, not the algorithm.
The corruption was not limited to cases that should be zero. Other incorrect outputs included:
- Int [(1.1)3] = 0, expected 1
- Int [(1.1)107] = 32809, expected 26854
- Int [(1.1)-3] = 1, expected 0
For the application, these errors translated into decompressed files with wrong sizes and no End-of-File (EoF) terminator. The result was dangling file nodes, missing rows, and no trail of corruption within the application itself. Such data-dependent, core-specific faults are computationally hard to detect without a targeted reproducer, particularly over a distributed fleet running millions of computations per second.
Hardening the stack
Once the reproducer was integrated into existing detection mechanisms, additional machines were flagged as failing the test. The incident drove changes on both fronts: hardware resilience and production detection were paired with more fault-tolerant software designs. The paper argues that no single layer catches silent corruption reliably—robust architectures must assume hardware can fail silently and be structured to absorb such errors without cascading data loss.
The full paper, Silent data corruptions at scale, details the root-cause flow and the multi-layered detection and mitigation strategies that emerged from this investigation.
