Why Dropbox Rethought Its Metadata Storage

Every Dropbox request depends on metadata—information about files, folders, and the services that power both internal and external products. That metadata now spans petabytes and is served from thousands of machines running sharded MySQL, handling tens of millions of queries per second with single-digit millisecond latency. The two main systems involved, the Filesystem and Edgestore, have served us well, but growth has exposed the limits of the architecture.

Edgestore originally ran directly on sharded MySQL, with each storage server holding multiple database shards. Data distribution was even, so most disks filled up at roughly the same rate. When capacity ran low, the expansion strategy was simple: split each machine in two, with each half taking ownership of a subset of the shards. By 2019, however, another split was on the horizon, and the cost of that approach was becoming prohibitive. Worse, there was a real risk that an outlier shard could grow beyond the capacity of any single machine, with no way to split it further.

The goal was to move away from doubling the fleet whenever we ran out of room and toward incremental capacity expansion based on projected growth. We also needed the ability to rebalance data across machines, since some shards naturally grow faster than others.

A New Storage Layer: Panda

The solution we settled on was to insert a new layer into the metadata stack: a petabyte-scale transactional key-value store called Panda, sitting between Edgestore and sharded MySQL. Panda abstracts away the MySQL shards, supports ACID transactions, enables incremental capacity expansion, and provides a unified implementation of features that multiple backends across Dropbox had been reimplementing separately.

In this new architecture, the bottom layer handles the heavy lifting of redundant storage, data rebalancing, and distributed transactions. The upper layer continues to offer higher-level features such as indexes, schema management, and strongly consistent caching. The clean separation means the team working on the application layer can focus on API improvements and new features for Edgestore, while the team responsible for Panda iterates independently on storage concerns.

A diagram of where Panda sits in our metadata stack—between sharded MySQL (or some future storage implementation) and systems such as Edgestore and the Filesystem.

Abstraction to the rescue

Because Panda sits above the storage engine, features like data rebalancing and two-phase commit can be implemented once in Panda rather than being tied to MySQL's capabilities. This also means we could, in principle, migrate metadata between entirely different storage engine implementations. For historical reasons, Edgestore and the Filesystem had diverged into separate backends; with Panda underneath, unifying them becomes possible—and with it, a single two-phase commit implementation for both.

Evaluating Existing Open Source Systems

Before committing to building Panda, we seriously evaluated whether an existing open source system could meet our requirements. We needed something that could scale to multiple petabytes, support data rebalancing, provide ACID transactions out of the box, and deliver a high rate of linearizable reads at low latency. Three candidates stood out.

FoundationDB

FoundationDB is a well-engineered open source transactional key-value store maintained by Apple. After a thorough investigation, we concluded that it would not scale to our needs without significant modification. A single FoundationDB cluster is designed to run with up to 500 storage processes, each single-threaded—meaning we'd be limited to roughly 500 processor cores. Our Edgestore and Filesystem clusters each use more than 10,000 cores. Extending FoundationDB to that scale would be a major undertaking.

There was also a hard limit on linearizable read throughput. FoundationDB relies on a centralized timestamp oracle to generate timestamps for transactions and linearizable reads. Batching helps scale this oracle, but only so far—we found a practical ceiling on the number of latest read requests the system could serve, regardless of additional hardware.

Finally, FoundationDB's failure handling did not match Dropbox's availability requirements. A failure of any node in the transactional system takes availability to zero until reconfiguration completes. For a system serving metadata at our scale, where failure detection is inherently imprecise, that level of fragility was unacceptable.

Vitess

Vitess, the MySQL clustering system originally developed at YouTube, has proven itself at massive scale—but its design assumptions didn't fit our workload. It is optimized for SQL applications that can tolerate stale reads, and neither applies to Dropbox. Both Edgestore and the Filesystem lack a SQL interface, and both require the most current data on every read.

Vitess also lacks production-grade support for ACID cross-shard transactions. Two-phase commit is only implemented as an experimental feature and is explicitly documented as not production-ready. Moreover, the implementation doesn't provide ACID isolation. Vitess argues that isolation can't be achieved efficiently, either through read-locks or multi-version concurrency control—a stance that would force us to abandon guarantees our engineers already rely on.

Fundamentally, Vitess embraces relaxed consistency. Its resharding process, for example, switches reads before writes complete, which can produce stale results. That's a reasonable trade-off for a video platform, but not for a file system, where reads must reflect all writes that completed before the read began.

CockroachDB

CockroachDB is the most mature of the NewSQL databases we examined, and its architecture looked promising. But at the time of evaluation, it had several blockers—some soft, some hard—that prevented adoption.

The hardest blocker was replication. Dropbox uses MySQL semi-sync replication between two regions separated by roughly 80ms of latency. This asynchronous model keeps read and write latency in the single-digit milliseconds while providing resilience against a regional disaster—at the cost of potentially losing writes that hadn't replicated to the other region in the event of a sudden outage. CockroachDB uses a quorum-based replication model that survives region outages without data loss but requires an 80ms inter-region write latency. Adopting that model would require years of data center buildout and operational changes, even if it were the right long-term direction.

Scale was another concern. CockroachDB's deployments at the time were, to our knowledge, more than an order of magnitude smaller than Dropbox's, and we weren't prepared to commit to a system that hadn't been regularly exercised at our operational scale—subtle implementation bottlenecks tend to appear with that kind of growth.

Memory usage was also a potential constraint. When we first tested CockroachDB, the default table range size was 64 MB. With a 2 PB dataset, that translates to over 30 million ranges, requiring roughly 60 GB of memory just to track them all. The range size is configurable, but larger values came with their own concerns. More recent versions have raised the default to 512 MB, which is a step in the right direction, but it wasn't sufficient for us then.

CockroachDB may well be viable for Dropbox in five to ten years as its deployments grow and remaining blockers are removed. For the time being, though, it wasn't ready for us.

A Purpose-Built Alternative

The open source landscape didn't offer a system that met all our requirements simultaneously. Building our own storage layer meant we could keep the consistency guarantees and operational model Dropbox engineers already depended on, while gaining the ability to scale capacity incrementally and rebalance data as needed. That's what Panda was designed to deliver: the abstraction boundary to separate storage concerns from application logic, the transactional guarantees our workload demands, and the flexibility to evolve the storage engine underneath without disrupting the metadata stack above.

Panda: A purpose-built metadata store

Taking destiny into our own hands. “Panda & Theseus” (2022) by Chia-Ni Wu is licensed under CC BY 4.0

Panda is a distributed, ordered key-value store that splits the key space into ranges and distributes those ranges transparently across storage nodes. It provides global ACID transactions through two-phase commit and hybrid-logical clocks, which are monotonic versions tied to real time. Beyond simple key-value pairs, Panda maintains a timestamped version history per key, enabling consistent point-in-time snapshots. The logical hierarchy mirrors a traditional relational database: database :: table :: (key, timestamp) -> value.

Data plane: routing and storage

The data plane handles all read and write requests through two node types: - Front end nodes serve as the system's entry point. They route requests to storage nodes and coordinate two-phase commits for ACID-compliant writes. The minimal API they expose is designed to support higher-level abstractions such as Edgestore and the Filesystem. - Storage nodes offer a lower-level API, including methods for two-phase commit. They interface with the underlying storage engine—MySQL in this case—to serve reads and writes. Read-only nodes serve reads once a snapshot is selected. Storage nodes implement multi-version concurrency control (MVCC) to support non-blocking reads and point-in-time queries. Critically, storage nodes do not communicate with one another on the data path; any pending writes they observe are bubbled up to the front ends for resolution. This keeps latency root-cause analysis straightforward with standard observability tooling.

Control plane: metadata and maintenance

The control plane orchestrates internal system processes through several subsystems: - Storage control is colocated with each storage node. It exposes low-level methods for range transfers, enabling multiple storage nodes to exchange ownership of data ranges. - Registry acts as the source of truth for all range metadata. It tracks which storage node owns which range, enforces invariants during range transfers—including exclusive write access—and guarantees full key space coverage. Front ends consult the registry to obtain up-to-date range-to-node mappings for request routing. - Keeper is the system's maintainer. It coordinates range transfers for data rebalancing and validates metadata properties like correct range placement on nodes.

Designing for multi-tenancy and predictability

Panda's API is deliberately minimalistic. It provides a reliable, predictable building block from which engineers can construct higher-level abstractions. The supported operations are intentionally basic: 
    type Panda interface {
      // If all Preconditions hold, atomically commits all KeyValue pairs at
      // the returned timestamp.
      Write([]KeyValue, []Precondition) (Timestamp, error)
    
      // Establishing latest (linearizable) snapshot.
      LatestTimestamp(Key) (Timestamp, error)
      LatestTimestampForRange(KeyRange) (Timestamp, error)
    
      // Snapshot read operations.
      Get(Timestamp, Key) (Record, error)
      Scan(Timestamp, KeyRange) ([]Record, error)
    }
The underlying principle: if a developer writes a query that takes down the database, that's the database's fault. Only a carefully curated, multi-tenant-aware API can uphold this standard. Key design tradeoffs include: - Predictability. Panda offers no higher-level read operations like joins. While this pushes complexity to developers, it lets Panda model operation costs precisely and reject traffic that cannot succeed. Performance degrades gracefully, batch jobs are throttled in favor of live traffic, and no single workload can exhaust system resources. - Partial results. All read APIs may return fewer results than requested. This guarantees incremental progress rather than timeouts or memory exhaustion on oversized requests—a practical defense against hot data regions that can otherwise stall entire processes. - Optimistic concurrency control. Key locks are not exposed. Instead, developers use preconditions to detect conflicting writes after their transactions begin. This may be less ergonomic, but it prevents any user from indefinitely holding resources and disrupting other transactions. - Data distribution. Developers influence placement only through key design. Structuring keys so co-accessed objects are close together increases the chance a request touches a single storage node, lowering write overhead and read latency. Importantly, transactional correctness is independent of physical placement; atomic read-write operations work regardless of which nodes hold the keys.

Why MVCC

Multi-version concurrency control was a foundational decision. Reads constitute over 95% of Edgestore and Filesystem workloads, and MVCC enables two capabilities that profoundly improve scalability. First, ACID transactions proceed without read locks. With read-dominated workloads, non-blocking reads dramatically increase throughput. Second, MVCC supports snapshot reads at older timestamps, allowing reads to be served from all nodes, not just write leaders. A typical setup has one leader accepting writes and recent reads, with two or more followers replicating data. Many Dropbox workflows establish a timestamp from a recent read, then perform many subsequent reads at that specific timestamp. Panda will even return an older timestamp when no recent writes have occurred, maximizing the chances those reads hit follower nodes. MVCC does carry costs. Write amplification is inherent—every operation, even deletes, becomes an insert. But the read-heavy workload makes this acceptable. Garbage collection of old versions is more subtle. Keeping too many versions for hot keys creates dense key ranges that degrade scan performance significantly. Panda addresses this by maintaining an ordered queue of keys and timestamps eligible for vacuuming, which bounds cleanup overhead.

Range transfers: flexibility at scale

Panda partitions keys into ranges of roughly 100 GB for Edgestore and Filesystem workloads—large enough to avoid unnecessary multi-node operations, yet small enough to move quickly when required.
A flowchart describing Panda’s range transfer protocol

Panda’s range transfer protocol

Transfer steps are engineered to shrink the critical section—when a range is unavailable or read-only—to seconds, even for ranges holding hundreds of gigabytes. The same protocol also enables range splits and merges, providing complete control over data distribution. This feature delivers several practical benefits: - Elastic capacity. Data growth by 5% requires only 5% more hardware. No more manual splits; consolidation is equally straightforward. - Version rollback. MySQL 8 made minor version upgrades one-way journeys. This is a significant operational risk given Dropbox's large MySQL fleet and its early exposure to upstream bugs. Range transfers let engineers revert any MySQL upgrade safely, regardless of future upgrade policies. - Alternative storage engines. Range transfers permit experimentation with different backends. A pilot implementation uses RocksDB as the storage engine with etcd/raft for replication. Transfers can move data between MySQL and this stack, providing a potential evolution path if no open-source alternative can satisfy Panda's API at Dropbox scale.

Verification strategy

Metadata stores that house Dropbox's users' critical data demand rigorous durability and consistency guarantees. Beyond standard unit and integration tests, Panda undergoes simulation testing: production-like workloads run for hours against a test environment while faults are injected and background processes—range transfers and MVCC garbage collection—operate concurrently. Verifiers continuously check core invariants: repeatable snapshots, linearizable latest reads, and consistent internal metadata like range placement. This stress testing has surfaced dozens of bugs, from deadlocks to range-transfer data corruption, swallowed errors, and even MySQL defects. A successful test run must also demonstrate meaningful throughput—a simulation that does no work has no value. Verification persists in production. A random sample of real user traffic is validated against the same invariants used in simulation. Production-only bugs—those requiring real workload patterns or enormous scale—inevitably exist, and this ongoing validation catches issues before they affect other Dropbox teams.

Lessons from building a distributed system at scale

Panda was a large undertaking and a central piece of Dropbox's long-term plan to modernize its metadata stack. That scale forced some deliberate engineering choices, and the project yielded several lessons worth passing along.

Every design decision carries a cost

Building a distributed system is essentially a series of trade-offs. MVCC brings clear benefits but adds write overhead and may not suit every workload. Abstraction layers decouple components and let you improve each independently, but the boundaries they create can block certain optimizations. There is no universally correct answer — the right call depends on what your specific requirements demand.

Offer the weakest guarantee that still works

Several times during Panda's development, the team defined APIs with stricter constraints than necessary. For example, the Change Data Capture API originally promised to deliver all updates for a single key in order, even when range transfers occurred. No initial Panda user relied on that behavior, though there were speculative future use cases. The stronger guarantee made the API harder to build and consume, and ultimately slowed progress. Starting with a weaker guarantee would have let the team ship Panda and strengthen the contract later if needed. The same principle led to read APIs returning partial results. Moving from a weak guarantee to a stronger one is always possible; the reverse is not.

Put production workloads in front of the system early

Shadowing production requests against Panda early on surfaced performance bottlenecks caused by incorrect assumptions about real workload patterns. In retrospect, an MVP should have been built and pointed at Edgestore traffic much sooner. That would have exposed unknown unknowns earlier and avoided time spent optimizing implementation details that turned out not to matter in practice.

Randomized testing earns its investment

Building solid randomized testing infrastructure requires substantial up-front effort. In this project, it paid for itself many times over — the team credits it as essential to shipping and sustaining Panda. If correctness is a priority, randomized testing is worth the cost.

You won't have every answer at the start

The future backend for Panda is still unknown, and the eventual answer might mean the project didn't take the shortest possible route. Large projects rarely follow a straight line. A sound multi-year technical strategy should deliver incremental value and leave room to adjust based on what you learn and how business needs evolve.

With Panda, expanding storage capacity incrementally is not only more cost-effective but a functional improvement over the sharded MySQL stack it complements. The abstraction layer gives metadata teams room to keep improving the underlying storage, which translates into cost savings for the business and performance gains for Dropbox users.