Why replication isn't enough
Dropbox's Magic Pocket storage system was built around a single, non-negotiable requirement: durability. The system employs a Reed-Solomon erasure coding variant similar to Local Reconstruction Codes to spread data across physically isolated hardware. A Markov model incorporating expected worst-case disk failure rates and repair times yields an estimated 27 nines of durability for a given block.
That theoretical number, however, only covers random hardware failures. Real-world durability demands protection against a broader, messier set of threats: software bugs, operator error, bad configuration changes, and natural disasters. Replication is necessary but not sufficient. To address this, the engineering team categorizes its defense-in-depth strategy across four dimensions: isolation, protection, verification, and automation. Verification alone accounts for a massive share of system activity—more than 50% of the workload on disks and databases is internal verification traffic.
An extent is the physical unit of data on a storage node: each node stores thousands of 1GB extents filled with user data blocks.
Scrubbing the disks
At the lowest level of the stack, the Disk Scrubber operates on the assumption that hard drives lie. With over half a million disks in production, failures range from bad sectors and silent bit-flips to fsyncs that don't actually persist. Many of these errors slip past S.M.A.R.T. monitoring and remain dormant.
This matters because the durability model assumes rapid repair. When a bad disk is detected, Magic Pocket re-replicates its data in under an hour. Undetected failures extend the vulnerability window from hours to months. The scrubber reads back every bit on each disk and validates against checksums. Bad data triggers automatic re-replication and sends the disk into a remediation workflow. A full sweep of each disk runs every 1-2 weeks using sequential scans to minimize seeks; recently modified areas receive more frequent checks.
Guarding deletions
When a volume must be moved or rewritten after garbage collection, Magic Pocket writes the new copy to fresh storage nodes before deleting the old one. That deletion transition is inherently risky—a software bug could erase an extent that was never stably rewritten.
The system mitigates this with a trash mechanism. When the Master instructs a node to delete an extent, the node merely moves it to temporary on-disk storage. The Trash Inspector then iterates over every block in these trash extents, consulting the Block Index to verify either that the block has been safely moved to a new node set or that the block itself was marked for deletion. Only after passing inspection does an extent remain for an additional day—a safeguard against bugs in the inspector itself—before being unlinked from the filesystem.
The Extent Referee provides a second layer for deletion safety. It watches each filesystem transition, ensuring any move or unlink corresponds to a successful trash inspection and explicit Master instruction. Any non-conforming transition raises an alert. Unix access controls and TOMOYO mandatory access control further restrict operators and unintended processes from touching storage node data.
Validating metadata and end-to-end reads
The Block Index lives in MySQL, which makes full table scans straightforward. The Metadata Scanner iterates through this index at roughly one million blocks per second, determining which nodes should hold each block and querying those nodes to confirm presence. Storage nodes keep sufficient in-memory metadata to answer these queries without disk seeks. The scanner and the Disk Scrubber work in tandem: one ensures disk contents match the Block Index, the other ensures the disk contents themselves are intact. The goal is a complete metadata sweep approximately weekly, providing confidence before code releases advance to the next storage zone.
Other verifiers guard against subtler failures. The Storage Watcher was written by someone outside the storage team specifically to avoid self-confirming bugs—the same broken assumptions that produce a faulty system often produce faulty tests for that system. This black-box checker samples 1% of all blocks written to Magic Pocket, records their storage key hashes in Kafka queues, and attempts to fetch each block after one minute, one hour, one day, one week, and one month, verifying end-to-end correctness over time.
The Cross-zone Verifier operates outside Magic Pocket entirely, scanning the separate Dropbox File Journal filesystem. It checks that all files are correctly stored in all zones for a given user. While other verifiers confirm Magic Pocket correctly stores known blocks, this verifier ensures agreement between what the system stores and what the rest of Dropbox believes should be stored—critical when moving users between zones or recovering from maintenance and outages.
Verifiers as a release gate
Building this verification stack was a significant engineering investment, but it pays off beyond production correctness. The same mechanisms serve as comprehensive integration testing for new code. A Staging cluster—tens of petabytes storing a geographically distributed mirror of production data—serves live traffic, falling back to production Magic Pocket on problems. Verifiers run for at least a week on Staging before team members can sign off on a code release into production. This gives the full suite a clear view into system correctness and catches bugs before they ever touch real user data.
Verifying the verifiers
How do you check that a verification system is actually working when all it produces is a flat line of zeros?
Magic Pocket (MP) is built for extremely high durability, so the primary metric from the Metadata Scanner — missing hashes — is consistently zero. Apart from regular disk errors caught by the Disk Scrubber and occasional timeouts from other verifiers, every verification graph looks the same: boring. That raises an obvious question: are the scanners genuinely detecting problems, or are they just idly reporting no issues day after day?
The answer is to deliberately create problems and confirm they get caught. This intersects with Dropbox's Disaster Recovery Training (DRT) events, which are induced failures designed to test both system recovery and team incident response. A DRT has a second, often overlooked purpose: proving that the verification mechanisms themselves are functioning.
Constructing failure tests that trigger verification without risking user data typically means running them in the staging cluster, which is safely backed up in production clusters. One notable test worked like this:
An engineer was given permission to secretly corrupt data in staging — flipping bits, truncating extents, and inserting random data. The team then waited to see if everyone detected the full set of errors. In that particular run, all errors were found quickly, except two that MP automatically repaired before the verifiers reached them. Crucially, the test caused no data loss in the staging cluster; MP's built-in redundancy detected and recovered from every failure without operator intervention.
The value of confidence
Building all of MP's verification systems was substantial work; few large-scale systems are subject to this level of scrutiny. Every project balances effort against correctness, and for Dropbox, data safety is paramount.
Regardless of where a project lands on that spectrum, confidence in correctness is empowering. Being able to reason about the current state of a storage system enables faster future development, more confident responses to operational issues, and reliable client applications free from confusion about the source of errors or inconsistencies. It also makes it easier to sleep at night — a worthwhile investment in itself.
Future posts will explore operational issues involved in running a system at this scale, followed by a discussion of the Diskotech architecture supporting append-only SMR storage.
