Increasing Magic Pocket write throughput by removing our SSD cache disks

A New Write Path for Magic Pocket

When Magic Pocket moved to shingled magnetic recording (SMR) drives in 2017, its storage hosts used SSDs as a write-back cache. Live writes landed on an SSD first, acknowledgements went back upstream immediately, and a background process later flushed those random writes to SMR disks as sequential writes. That design let Dropbox take advantage of higher disk densities while preserving durability and availability guarantees.

Over time, that architecture created a bottleneck. Each storage host could hold more than 100 data disks (14–20 TB each, for 1.5–2 PB of raw data per host), but all of those disks shared a single SSD for live writes. A host’s maximum write throughput was capped by the write throughput of that one SSD—even NVMe drives, which can handle 15–20 Gbps, could not keep pace with the cumulative throughput of hundreds of disks. As disk density grew, the problem got worse: fewer hosts meant each remaining host’s SSD had to absorb more writes.

In 2021, Dropbox decided to remove the SSD cache entirely and write directly to SMR disks. The payoff: a 15–20% increase in write throughput across the fleet, better reliability, and lower storage costs.

Why the SSD Cache Had to Go

Throughput was not the only pressure. In early 2020, a large group of SSDs failed within a short window. The drives had been added to the fleet around the same time, followed similar write patterns, and reached their write endurance limits together. Magic Pocket was forced to perform a massive data repair all at once—a serious durability risk that the system ultimately survived, but only after significant effort.

There was also operational overhead. Every new server hardware generation required SSD qualification for reliability, durability, and efficiency, and the capacity and supply chain teams had to project SSD demand. Removing SSDs would eliminate a single point of failure, reduce infrastructure complexity, and cut hardware and repair costs. The team started exploring that possibility during Hack Week in 2021.

Inline Metadata on SMR Disks

Magic Pocket organizes data into extents—containers of blocks, typically 1–2 GB. An open extent accepts writes until it reaches capacity, then closes and becomes immutable. Because extents were already mapped to a fixed set of SMR zones and written sequentially, the storage engine’s extent model fit SMR’s sequential-only write constraint well.

The key change was in how extents store data. Previously, block metadata (length, hashes, and similar fields) was written to the SSD cache while raw block data went to SMR disks. To eliminate the SSD, the team introduced a new extent format that stores both metadata and raw data inline on the same SMR disk. On startup, the storage engine parses the extent into metadata and data, then builds an in-memory index mapping each block to its offset. For new writes, blocks are serialized with their metadata and committed directly to the SMR disk.

Control plane workflows also changed. Disk provisioning previously formatted, partitioned, and assigned SSD caches to SMR disks; failure detection workflows flagged SSD failures for repair. With SSDs gone, those operations were no longer necessary.

Testing, Tradeoffs, and Results

Before rolling out the changes, the team built an automated tool to simulate crashes, partial writes, and bit rot, verifying that the storage engine could recover cleanly in every scenario. The new extent format was also validated against background traffic before full deployment.

Load tests under peak throughput showed a 2–2.5x increase in write throughput and a 15–20% improvement in p95 write latency. In production, where load is spread across all hosts and disks rather than pushed to peak, the numbers were different. Average-load p95 disk write latencies rose 10–15% because SMR writes are inherently slower than SSD writes. That translated to only a 2% increase in end-to-end Magic Pocket write latency, a tradeoff the team accepted.

Read latency was another consideration. The storage engine previously kept recently written blocks in an in-memory cache to cover the window before data was flushed from SSD to SMR. Removing the SSD meant removing that cache, which raised latency for reads of recently written blocks—but p99 latencies showed no significant impact. If needed, the team noted, a block cache could be added back later.

Production Rollout

The feature began with a subset of Magic Pocket hosts and expanded as confidence grew. By the end of Q1 2022, every SSD had been removed from Magic Pocket, and storage hosts were writing directly to SMR drives.

The results went beyond throughput. The fleet-wide write throughput improved 15–20%, helping Magic Pocket keep pace with growing workload demands without adding hosts. The drives’ failure modes also improved: previously, 8–10 SSDs failed per month, each requiring manual inspection and replacement by the data center operations team. Removing the SSD cache eliminated that failure point and reduced repair volume. Hardware platform design is also simpler with one fewer component to qualify and integrate.

The project delivered gains in reliability, efficiency, and cost across Dropbox’s storage infrastructure. For Magic Pocket, writing directly to SMR disks proved that the SSD cache had become an unnecessary middleman—and that removing it could unlock both throughput and operational simplicity.