Why Dropbox Built a Metadata Service

Dropbox first introduced Edgestore, its internal metadata store, in late 2013. Since then, the system has grown from a client-side ORM wrapper into a full service layer that supports a range of Dropbox products. Today, Edgestore runs across thousands of machines in multiple data centers, storing trillions of entries and serving millions of queries per second with five-nines availability.

From Sharded MySQL to a Unified Store

Dropbox began with vanilla MySQL for metadata, but rapid growth in users and features produced a sprawling set of independent databases that frequently needed sharding. That architecture created several problems:

  • Developers spent time writing raw SQL; schema evolution was brittle and error-prone.
  • Operating many independent MySQL hosts demanded significant operational effort.
  • Reactive resharding at capacity limits was cumbersome, and spinning up new clusters for every use case carried heavy overhead.
  • Without isolation, a slow query or bug could degrade performance everywhere.

By late 2012, it was clear that the next system needed to support a broad class of workloads, offer developer-friendly APIs and easy schema evolution, and still meet the durability, availability, and performance expectations of critical infrastructure. No off-the-shelf database satisfied those requirements. Drawing on in-house MySQL expertise and the design of similar systems—notably Facebook's TAO—Dropbox decided to build a service that hid the database behind higher-level abstractions while keeping MySQL (InnoDB) as the storage engine.

What began as a simple client-side ORM has since become a standalone service with caching, geo-replication, and multi-tenancy baked in.

How Edgestore Is Put Together

Developers interact with Edgestore through Go and Python SDKs that implement the Edgestore API. The API lets teams model their data without caring where or how it is physically stored. Data objects come in two forms: Entities and Associations, each of which can carry user-defined attributes analogous to columns in a relational table. Associations express relationships between entities—for example, UserEntity, TeamEntity, and a UserTeamAssoc can represent group membership.

Requests flow from SDK clients into one of several Edgestore Cores, a stateless routing layer that forwards each request to the correct shard or region. Because Dropbox workloads are read-heavy, Cores sit in front of a partitioned and replicated Caching Layer. Writes invalidate cache entries, preserving strong consistency by default; callers that can accept stale reads may opt into eventual consistency.

On a write or cache miss, a Core passes the request to the Engine layer for the shard where the data is mastered. Engines translate Edgestore API calls into MySQL commands and abstract away the storage backend, keeping the design storage-agnostic. Engines also meter resource consumption by traffic source, which is the basis for isolation and multi-tenancy across all workloads sharing the cluster.

Representative Workloads

Edgestore's flexible data model accommodates graph structures, key-value stores, and queues in production. It backs some of Dropbox's most visible features.

Shared Folders

Collaboration features are modeled as a graph: users and folders are entities, and bidirectional associations make it straightforward to list every folder a user can access or every user with access to a given folder. Revoking a user's access is a matter of removing a single association.

Email Campaigns

An internal email management system leverages Edgestore as a queue. Each message slated for delivery maintains an association with its current status—pending, scheduled, sent, or errored—plus a timestamp for ordering. A periodic job scans timestamps and statuses, sends due messages, and removes them from the queue or re-enqueues them when a retry is needed. All of that metadata lives in Edgestore, and the strong consistency guarantee is what gives the system exactly-once delivery semantics.

U2F Security Keys

Dropbox's support for Universal 2nd Factor (U2F) security keys depends on Edgestore as well. The service stores hardware token metadata, the public key bound to each token, and related data in an association between a User entity and a Host entity.

What's Next for Edgestore

Edgestore is a strongly consistent, read-optimized, horizontally scalable, geo-distributed metadata store. Rather than layering more tooling onto direct MySQL access, Edgestore abstracts the database away entirely so product teams can move quickly without taking on operational burden.

Dropbox plans to publish deeper technical posts in the coming months. Planned topics include:

  • Data Model and Schema Evolution: How developers express schemas, how those schemas are represented on disk, and how they change over time.
  • Consistent Caches: Keeping in-memory caches aligned with disk state under strong consistency, without sacrificing availability or performance.
  • Event Stream: An abstraction that lets applications react to metadata writes and updates as they happen.
  • Cross Data-Center Routing and Replication: The custom replication pipeline, built on the event stream, plus the request routing and forwarding logic that enables geo-replication.
  • Multi-Tenancy: Attribution and isolation across dozens of internal and external workloads sharing a single Edgestore deployment.
  • Operations and Tooling: Managing a mission-critical, highly available distributed service in production.