Inside the Magic Pocket

Architecture at a Glance

Magic Pocket (or just “MP” internally) is Dropbox’s in-house storage system for file content, built to hold multiple exabytes of immutable blocks. It handles encrypted chunks of files up to 4MB in size, with no in-place edits once data is written. This design choice pushes mutability concerns up the stack—FileJournal tracks file changes separately, keeping the storage layer simple.

The workload is marked by high temporal locality: most blocks are read heavily within the first hour after upload, then accessed less frequently but still with low-latency requirements. To match that pattern, MP leans on spinning media—hard drives—for their durability, density, and cost, while reserving SSDs for databases and caches. Recent uploads take advantage of generous replication and caching, while older data shifts to a more storage-efficient erasure coding scheme.

Design Drivers

Durability is treated as absolute. The goal is effectively infinite data persistence—random disk failures should be less likely than a catastrophic asteroid event. Data is erasure-coded and spread across multiple geographic regions to guard against both hardware faults and natural disasters.

Scale was a central challenge from the start. MP had to grow from double-digit petabytes in prototypes to multi-exabytes within roughly six months. That pace forced careful bottleneck elimination during design and an architecture flexible enough to adapt as new constraints appeared, such as sudden traffic spikes that saturated inter-cluster routers. In response, data placement and request routing were reworked to respect cluster affinity, and the inter-cluster network topology was eventually revisited as well.

Simplicity is a deliberate virtue. MP avoids quorum-based consensus and distributed coordination whenever possible, preferring fault-tolerant centralized coordination points. For the Block Index, that meant choosing a sharded MySQL cluster over a distributed hash table or trie—a decision that simplified development and reduced unknowns across the system.

Data Model

MP stores opaque, encrypted blocks, each up to 4MB:

Blocks are compressed and encrypted before passing to MP. Their identifiers are typically SHA-256 hashes of the block content.

Individual blocks are too small to move efficiently as units, so they are grouped into 1GB logical containers called buckets. Buckets are time-based aggregations—blocks uploaded around the same time land together regardless of content. For reliability, buckets are replicated across physical machines. New blocks are first replicated directly, then the containing buckets are aggregated and erasure-coded to improve storage efficiency. A volume is the term for one or more buckets replicated across a set of storage nodes.

How Magic Pocket Is Put Together

Magic Pocket uses a multi-zone setup, with server clusters in the western, central, and eastern United States. Every block is stored independently in at least two zones and then reliably replicated within those zones. Beyond protecting against natural disasters and large-scale outages, this design creates clear administrative domains and abstraction boundaries, so a misconfiguration or congestion collapse in one zone can't cascade into another.

Most of the complexity lives inside a single zone. A zone is built from these primary pieces:

Frontends

These nodes are the gateway to Magic Pocket and accept all storage requests from the outside world. They decide where a block should live and issue the internal commands needed to read or write it.

Block Index

The Block Index maps each block to the bucket holding it. Underneath, it's a massive sharded MySQL cluster fronted by an RPC service layer, with substantial tooling for database operations and reliability. Dropbox originally planned to build a dedicated key-value store here, but MySQL turned out to be more than capable. With thousands of database nodes already in service across the stack, the team could reuse its operational experience managing MySQL at scale.

hash → cell, bucket, checksum

The real schema is a bit more involved to handle deletes, cross-zone replication, and similar concerns. Key-value stores are fashionable and fast, but databases offer high reliability with an expressive data model that has made it easy to grow the schema and functionality over time.

Cross-Zone Replication

A dedicated daemon asynchronously replicates all block puts from one zone into another. Any block uploaded locally is written to a remote zone within one second of being stored. That replication delay is factored into the durability models, and data is always replicated widely enough locally to cover the gap.

Cells

Cells are self-contained logical storage clusters holding roughly 50 PB of raw data each. Adding capacity to Magic Pocket generally means bringing up a new cell. Although cells are logically independent, each one is striped across the racks so that physical diversity within a cell is maximized.

Inside a Cell

Object Storage Devices (OSDs)

OSDs are the workhorses: storage boxes full of disks, with a single machine able to hold over a petabyte and a single rack over 8 PB. They contain complex logic for caching, disk scheduling, and data validation, but the rest of the system treats them as “dumb” nodes. They store blocks without understanding cell topology or taking part in distributed protocols.

Replication Table

The Replication Table is the cell's index, mapping each logical bucket of data to the volume and the OSDs where that bucket is stored. Like the Block Index, it lives in MySQL, but it's much smaller and updated far less often. The working set fits entirely in memory, producing very high read throughput from only a few physical machines.

bucket → volume
volume → OSDs, open, type, generation 

Two fields in the schema matter especially. The open flag states whether a volume is open for new writes (open) or immutable (closed); only a small number of volumes are ever open at once. The generation number keeps operations consistent when volumes are moved to recover from disk failures or optimize layout.

Master

The Master is the cell's coordinator. It contains most of the complex protocol logic, watches OSD health, and triggers data repair when disks fail. Background operations also run through it: creating new buckets when old ones fill, garbage collection after deletes, and merging buckets that shrink too far.

The Master is entirely soft-state; all authoritative volume information lives in the Replication Table. It is not on the data plane either. No live traffic passes through it, so reads continue even if the Master goes down. Writes can also proceed for a while, though the cell will run out of open storage buckets without the Master to create new ones. Other cells always remain available for new writes in the meantime.

A single Master runs per cell, giving a centralized point for placement decisions and sparing the team the considerable complexity of a distributed protocol. That central model does cap cell size: roughly a hundred petabytes before memory and CPU become bottlenecks. Multiple smaller cells are convenient from a deployment view too, and they give better isolation against cascading failures.

Volume Managers

These processes handle heavy lifting, responding to Master requests to move or erasure-code volumes. That usually means reading from many OSDs, writing to others, then passing control back to the Master. Volume Managers run on the same physical machines as OSDs so their heavy network-capacity requirements can be spread over otherwise idle storage hardware.

Core Protocols

Put

Frontends know the storage state before a Put arrives: they periodically contact each cell for available space and maintain a list of open volumes that can receive new data.

When a Put request lands, the Frontend first checks the Block Index to see if the block already exists, then picks a target volume. Volume selection balances cell load and reduces network traffic between storage clusters. The Frontend consults the Replication Table to find which OSDs hold the volume, then sends store commands to those OSDs. Each fsyncs the block to disk or onboard SSD before responding. On success the Frontend adds a Block Index entry and returns to the client. If any OSD fails, the Frontend retries with another volume, possibly in a different cell; if the Block Index fails, the request goes to the other zone. The Master later cleans up partial writes from failed operations in the background.

A quorum-based protocol would let the Frontend write to just a subset of OSDs in a volume, cutting retries and possibly lowering tail latency — but at the cost of considerable complexity. Careful timeout management in the retry approach already produces tail latencies the team is happy with.

Get

Serving a Get follows naturally from the Put path. The Frontend looks up the cell and bucket in the Block Index, finds the volume and OSDs in the Replication Table, and fetches the block from one of those OSDs, retrying if needed.

Replicated data is straightforward: every OSD in the volume has every block.

Erasure-coded volumes need more care. Encoding is arranged so each block can be read whole from one OSD, meaning most reads touch just one disk spindle. That keeps hardware load low. When the selected OSD is unavailable, the Frontend reconstructs the block from encoded data on the remaining OSDs with help from the Volume Manager.

In the encoding scheme above, Block A is normally read from OSD 1 (green). If that read fails, the block can be rebuilt from sufficient data on the other OSDs (red). Real encoding schemes are more intricate and tuned to reconstruct from a smaller subset of OSDs under common failure scenarios.

Repair

The Master's most important job is Repair — re-replicating volumes after a disk fails. OSD health is monitored via the service discovery system, and a repair triggers once an OSD has been offline for 15 minutes. That window is long enough to restart a node without needless repairs, yet short enough for rapid recovery and minimal vulnerability.

Volumes are scattered throughout a cell, with each OSD holding several thousand volumes. Losing one OSD means its volumes can be rebuilt from hundreds of other OSDs at once.

With OSD 3 gone above, volumes A, B, and C can be recovered from OSDs 1, 2, 4, and 5. In practice thousands of volumes per OSD are shared across hundreds of peers, so reconstruction traffic spreads over hundreds of network cards and thousands of spindles, keeping recovery fast.

The Master first closes all volumes on the failed OSD and has the other OSDs record the change. Closed volumes are immutable and safe to move. Then it builds a reconstruction plan, choosing copy-source and replicate-target OSDs to spread load evenly — avoiding hot spots on any particular disk or machine. This planning is why spare capacity per OSD can be provisioned leanly, and it benefits directly from the Master's central coordination.

Data transfer runs through the Volume Managers, which copy, erasure-code where necessary, and hand control back.

The final step is simple but critical. The volume now exists at both source and destination, but the move is not committed. If the Master dies now, the volume simply stays put and the replacement Master repairs it again. To commit, the Master increments the generation number on the new OSDs' volumes, then updates the Replication Table with the new volume-to-OSD mapping and generation. This commit point makes lost updates impossible, even if the failed OSD returns to life.

The protocol stays consistent no matter which node dies at which point. Production has shown all manner of odd failures — a frozen database frontend that suddenly woke up an hour later forwarding a stale request, while a restarted Master had already issued a different set of repairs. The open/closed volume distinction is what keeps live traffic from interfering with background operations, enabling far simpler consistency protocols than would otherwise be necessary.

Keeping Distributed Storage Simple

Magic Pocket is a large distributed storage system, but its guiding design principle is deliberately modest: keep it simple. Building a storage system is difficult, but operating one reliably at scale is harder still. That requires not just the core software but the monitoring, verification, and tooling to ensure it runs correctly day after day.

A key part of the team's approach was choosing technical solutions that fit the actual problem, rather than adopting something because it was novel. The system was built by a team of fewer than six people, which forced a focus on what truly mattered and contributed significantly to the project's success.

Much of the fine-grained detail about operating at this scale is intentionally left for future discussion. The specific failure modes, edge cases, and operational nuances that come with running Magic Pocket are topics the team has considered in depth and will cover in dedicated posts about building and running systems at this level.