Why Dropbox Split Its Storage Into Warm and Cold Tiers

Magic Pocket, Dropbox's in-house multi-exabyte storage system, was built with a simple observation in mind: file access patterns are heavily skewed toward recent uploads. Data shows that over 40% of all file retrievals are for files uploaded in the last day, and more than 70% come from the last month. This means the vast majority of what's stored is rarely accessed — exactly the kind of workload where storage costs can be optimized.

The system has always leaned toward "cold" operation. It runs on spinning disks rather than SSDs, uses n-way replication for freshly uploaded data, and transitions older data into a more efficient erasure-coded format in the background. But there was one significant inefficiency: to guarantee durability against full regional outages, Magic Pocket kept complete, independently replicated copies of every block in at least two geographically separated regions.

Storage Tiers and the Efficiency Problem

The solution, internally, was to split storage into two tiers. The original Magic Pocket system became the "warm" tier, handling all frequently accessed data. A new "cold" tier was designed for data that's rarely read, with migration happening asynchronously in the background whenever files age out of the warm profile.

Cold storage had strict requirements:

  • Durability is non-negotiable. The cold tier must tolerate a full region outage plus multiple simultaneous rack failures in surviving regions — matching the original system's guarantees.
  • Read availability must remain high. Cold data may be rarely needed, but when a user requests it, the data must be available immediately, not in minutes.
  • Write availability is flexible. User-facing writes always go to the warm tier first, so writes to the cold tier can be paused at any time without affecting end users.

The obvious route to reducing overhead was to drop the full cross-region replication. If cold data were replicated across regions instead of fully inside each one, storage cost would drop. The tradeoff is higher wide-area network usage during a region outage, but for infrequently accessed data, that's an acceptable cost.

Two Failed Approaches

Single Erasure Code Across Regions

The first serious attempt was to treat all regions as one logical entity. Instead of each region independently replicating data, a single erasure code would span regions, with redundancy built in to survive a large-scale disaster. The prototype ran in staging quickly, but the team soon uncovered escalating complexity and hidden costs.

The deeper problem was durability risk. On paper, the design provided the same statistical "nines of durability." In practice, all three regions would run the same software version, and a single large-scale bug could potentially wipe out data regardless of redundancy. The independent region model is a powerful defense against human error — a faulty deployment or a bad operator command — and that protection would be lost entirely.

After more than nine months of active development, the project was cancelled.

XOR Buddy Pairs

Go back to the drawing board, the team reconsidered ideas previously ruled out. One that stood out was the approach in Facebook's Warm BLOB storage system.

The concept pairs each block with a buddy block in a different geographic region. An XOR of the two is stored in a third region. If block A is unavailable in Region 1, it can be reconstructed from block B in Region 2 and the XOR of both in Region 3. None of the underlying region internals would need any change — the different replication pattern simply builds a new layer on top.

On paper, that's an elegant trick. In practice, managing a globally available set of buddy pairs proved cumbersome. Unpredictable delete patterns required a space reclamation process to track which blocks were no longer needed, and all that extra coordination made the design too complex for Dropbox's use case.

Boiling down both failed attempts reveals a recurring theme: removing full replication inside regions introduces cross-region coordination, and that coordination itself creates new classes of failure and complexity. The challenge wasn't just data placement — it was how to fail independently, avoid global state, and still protect against human error.

Fragmentation over a single-region fetch

The failed experiments were not wasted effort. They sharpened our understanding of the network-stack tradeoffs and eventually led us to question a core assumption we had carried since the beginning: did a block read really need to be servable from a single region under normal conditions? For cold data, accessed infrequently, always paying a cross-region network cost is acceptable. Dropping that requirement opened the door to a much simpler design.

In a three-region example using XOR for parity — generalizable to more regions with Reed-Solomon erasure codes — the idea works like this. Instead of pairing similar-sized blocks from different regions, split a block into fragments and stripe them across regions.

To put a block, split it into two fragments. Store the first in Region 1, the second in Region 2, and compute an XOR parity fragment for Region 3. Since migration to the cold tier is asynchronous, we can simply pause it if a region is unavailable and resume when all three are back.

To get a block, send requests to all three regions, wait for the fastest two responses, and cancel the third. There is no preference over which two fragments arrive — XOR or erasure decoding is cheap compared with disk I/O and network transfer.

Deletion is equally straightforward: when a block is no longer referenced, delete all fragments. This can proceed independently per region, letting us run different software versions and stagger deletions so a single bug cannot wipe every copy.

Latency: one extra round trip, better tails

The obvious cost is that even the best-case read needs fragments from multiple regions. The measured impact was smaller than expected, and in one percentile it vanished entirely:

The 5th percentile in the cold tier is noticeably higher, as expected, because at least one cross-region round trip is unavoidable. From the 25th through the 95th percentile, however, the gap to the warm tier holds steady at roughly a single network round trip. Dropbox's network stack is built for transferring large blocks over long distances: a tuned gRPC-based RPC framework called Courier multiplexes requests over HTTP/2, keeping TCP connections warm with large windows so a multi-megabyte block can travel in one round trip.

The 99th percentile surprised us: the cold tier showed lower tail latency than the warm tier. The reason is that the cold tier takes the fastest two of three responses, while the warm tier's retry logic was more sequential — fetch from the closest region first, and only ask the second region if the first fails or times out. The warm tier sees most traffic, so making two parallel requests unconditionally would be wasteful. Instead, we modified it to optimistically fire a request against the second region if the first has not answered within a time budget. Tail latency dropped with only a small network overhead — a pattern described in Jeff Dean's Velocity talk.

A pleasing side effect of the cold tier's design is that it always exercises the worst case. There is no plan A versus plan B; a read always reconstructs from fragments. A region outage causes no traffic shift or disk-I/O spike in the surviving regions, unlike the warm tier or earlier designs. That removes a whole class of capacity-planning worry for emergency failover.

We had no fixed latency target for cold storage — only directional guidance — and the results beat expectations. The difference is small relative to internet transfer times, so end-user experience is unaffected. That allowed us to be more aggressive about what data we classify as cold.

Durability and cost

The durability win comes from keeping each region operationally independent with layered architecture. Running different software versions per region means a bug that escapes testing is unlikely to corrupt the same data in two or more regions simultaneously — a valuable defense against unknown unknowns, even if it is hard to quantify. It also lets us move faster without compromising durability.

There is still exposure to replication-logic bugs before data lands in all regions. But the surface area is limited, and we mitigate it by not purging the warm tier immediately after migration. Multiple rounds of validation run before any data is removed from its original location.

Cost savings come straight from the replication math. The warm tier uses a 1+1 scheme — one data fragment and one parity fragment — plus intra-region replication, for a total factor of 2 * region_internal_replication_factor. The three-region cold tier uses a 2+1 scheme, giving 1.5 * region_internal_replication_factor — a 25% reduction in disk usage. Generalizing to four regions with a 3+1 model yields 33% savings, and further extensions such as 4+1 or 5+2 are possible depending on the desired tradeoff between cost and tolerance for simultaneous region loss.

Constraints, intuition, and course correction

This project reinforced how much constraints shape problem-solving. Adding a firm constraint — independent failure domains — narrowed the search and ruled out dead ends early. Removing one — single-region reads at steady state — opened a new design space entirely.

It also showed that intuition is not always right. We rarely have complete data, and sometimes we do not know which data matters. The only way forward is to try, fail honestly, and course-correct. A bad decision is survivable if you are willing to step back and change direction.

The result is a cold storage tier that fits our needs: meaningfully lower storage costs with no reduction in durability or availability and no significant added architectural complexity.