One service to route them all

Dropbox has long run a mix of storage systems: Amazon S3, the Hadoop Distributed File System (HDFS), and its own internal block store, Magic Pocket. After migrating user file data to Magic Pocket in 2015, the company kept S3 and HDFS as general-purpose blob storage for crash traces, build artifacts, test logs, and image caches. That arrangement became increasingly costly—S3's per-request pricing punished workloads with many small writes, and caches built on S3 generated expensive GETs on every miss.

Rather than simply picking a cheaper vendor, Dropbox built a service in 2020 to sit in front of multiple storage backends. Named Object Store, the service exposes a single API and routes each request to the most cost-efficient backend. The layer also centralizes features like encryption, retention policies, and monitoring, and it allows traffic to be shifted between providers without data migrations. Dropbox credits the service with millions of dollars in annual savings.

A simplified S3 API with batched writes

Object Store is an abstraction, not a storage engine. It tracks object placement in a MySQL database and forwards data to backends such as S3 and Magic Pocket. The API mirrors a simplified S3 interface—PUT, GET, DELETE, and LIST—with access segregated by "pails," containers analogous to S3 buckets that share configuration and ACLs.

The core cost optimization is write batching. On a PUT, the request-servicing layer enqueues the blob in memory alongside other pending PUTs to the same pail. When the queue times out or reaches an aggregate size threshold, all queued requests are concatenated into a single batched blob and written to persistent storage. Object Store then records two kinds of metadata in MySQL:

  • A batch row, noting the batched blob's key in the persistent store
  • One or more object rows mapping each original object key to start and end offsets within the batched blob, plus a reference to the batch row

GETs work in reverse: the service fetches object and batch metadata, then performs a ranged read on the batched blob using the stored offsets. This collapses many S3 PUTs into one and avoids GET requests entirely on cache misses, directly attacking the two biggest line items in an S3 bill. When the batch is written to Magic Pocket—which runs most efficiently with homogeneous 1–4 MB objects—batching also shapes traffic into the size and rate profile that backend prefers.

Encryption at two levels

Object Store encrypts all data itself, independent of the underlying store. Each write involves two encryption steps:

  • The raw object blob is padded and encrypted with AES-256 using a unique 256-bit block encryption key (BEK) per object.
  • The BEK is then wrapped with a versioned global key encryption key (KEK), and the encrypted BEK is stored as part of the object metadata in MySQL.

This two-level scheme means data can only be read with all three components: the encrypted blob, the encrypted BEK, and the KEK. It also makes key rotation practical—rotating KEKs only requires decrypting and re-encrypting metadata, not rewriting stored object data. Dropbox rotates KEKs regularly as a security measure.

Deletion via crypto-shredding

Batching complicates deletion, especially when data must be purged from infrastructure by a deadline rather than merely made unreadable. An early design batched deletes as well: Object Store marked object metadata as deleted and ran a walker to purge fully-dead batches from S3. A second compaction walker handled batches where some objects were deleted but others still lived—breaking them down, merging the live objects into fresh batches, and rewriting everything.

That approach worked but was hard to maintain. Compaction bugs required forensics to undo, and the compaction walker itself generated PUTs and depended on regular, timely runs. Per-object encryption enabled a simpler path: wipe the object metadata. Without the BEK, the encrypted data in persistent storage is effectively unrecoverable. This technique—crypto-shredding—makes DELETE a synchronous MySQL operation with no deadline management and no compaction. Space reclamation in the backend then proceeds asynchronously at a slower cadence.

Chunking for large objects

The write-side batching model solved problems for small objects, but Magic Pocket's 4 MB optimal object size made storing larger objects infeasible. Object Store addresses this with chunking: objects over 4 MB are split into 4 MB chunks stored separately in Magic Pocket. At read time, the component chunks are fetched in parallel and reassembled. The system avoids extra metadata by storing a single base key and deterministically computing each chunk's key from it.

This optimization, completed at the end of 2021, made Object Store a general-purpose system that accepts data of any size and repackages it for optimal backend storage. With it, Dropbox deprecated the majority of its HDFS usage, realizing another round of multi-million-dollar annual savings.

Roadmap

Object Store is positioned as a strategic layer for Dropbox's storage infrastructure, letting the company shuffle data across providers in response to pricing and compliance changes. Although originally designed for non-file data (raw Magic Pocket remains the primary store for user file blocks), Dropbox plans to migrate subsets of user file data to Object Store. Planned work includes:

  • Lambda function support for object writes and modifications
  • Using Object Store as a backend for storage systems like Cassandra or RocksDB, enabling diskless compute hosts with an exabyte-scale remote disk
  • Backing third-party systems with pluggable object storage, such as Apache Pinot and Loki