A replication-first approach

Before ZippyDB, teams across Facebook ran RocksDB in production directly, each solving the same problems independently: consistency, fault tolerance, failure recovery, replication, and capacity management. The service was built to consolidate those workloads onto a single, highly durable key-value store that offloads data management at scale to a shared platform.

The central architectural decision was reuse over reinvention. ZippyDB layers a general-purpose replication library called Data Shuttle on top of RocksDB, then relies on existing infrastructure—Shard Manager for shard placement and load balancing, and a ZooKeeper-based configuration service for service discovery—rather than building those components from scratch.

Service topology and sharding

Deployments run as tiers, each consisting of compute and storage resources spread across multiple geographic regions. Today there are only a handful of tiers: a default "wildcard" multitenant tier plus specialized tiers for workloads such as distributed filesystem metadata. Most use cases land on the wildcard tier for better hardware utilization and lower operational overhead; dedicated tiers are reserved for stricter isolation requirements.

Each use case's data splits into physical shards, the server-side unit of data management, which typically hold 50–100 GB and host tens of thousands of micro-shards (μshards). Clients never address physical shards directly. μshards give ZippyDB the ability to reshard and move data transparently. Two mapping schemes are supported:

  • Compact mapping is used when the assignment is fairly static, changed only when a shard grows too large or hot—an infrequent event in practice.
  • Akkio mapping delegates μshard placement to the Akkio service, which locates data in the regions where it's most accessed, reducing data-set duplication compared with replicating everywhere.

Replication and consistency configuration

Data Shuttle is responsible for both synchronous and asynchronous replication. Within each shard, a subset of replicas forms the Paxos quorum group, or global scope, and data moves synchronously among those replicas using Multi-Paxos. Additional replicas can be configured as followers, analogous to Paxos learners, receiving data asynchronously. Followers let applications keep in-region replicas for low-latency reads with relaxed consistency while keeping the quorum small to hold down write latency. Applications can also specify stickiness constraints that hint where shard replicas should be placed, gaining some direct control over read and write latency.

Shard Manager assigns an epoch and a leader for that epoch for each shard. The leader holds a lease for the lifetime of the epoch, renewed by periodic heartbeats, and assigns a monotonically increasing sequence number to every write. Writes go to a replicated durable log via Multi-Paxos as the leader's total ordering is determined; once the ordering reaches consensus, entries drain in order across all replicas. When a failure is detected, Shard Manager assigns a new leader with a higher epoch, restoring write availability.

The current design uses this external service for leader assignment and failure detection, which simplified the initial implementation. Plans are to move to in-band failure detection inside Data Shuttle to reelect leaders more proactively and avoid waiting on Shard Manager.

Read and write semantics

Consistency and durability are not fixed properties of a tier or use case, but request-level options on the read and write APIs. The default write path persists on a majority of Paxos logs and writes to RocksDB on the primary before acknowledging. In fast-acknowledge mode, writes are acknowledged as soon as the primary enqueues them for replication, accepting lower durability for better latency.

The three most common read levels are:

  1. Eventual, which in practice is stronger than classic eventual consistency. Total ordering and a configurable lag threshold (enforced via heartbeats) make it closer to bounded staleness.
  2. Read-your-writes, where the client caches the latest sequence number from write responses (per client process) and issues at-or-later queries.
  3. Strong, where reads route to the primary, which relies on its lease to guarantee no other primary exists. If the lease renewal hasn't been observed, the read degrades to a quorum check.

Data model and APIs

ZippyDB exposes a minimal key-value API—get, put, delete with batch variants—plus prefix scans, range deletes, and a test-and-set interface for read-modify-write operations. Transactions and conditional writes are also supported for more generic read-modify-write workflows. Native TTL support lets clients specify expiry on write; cleanup piggybacks on RocksDB's periodic compaction while reads filter expired keys between compaction runs.

Many applications don't call ZippyDB directly but go through an ORM layer that translates higher-level accesses into ZippyDB API calls, abstracting the details of the storage service underneath.

ZippyDB also offers an optional caching layer and integration with a pub-sub system for mutation subscriptions on shards, both opt-in per use case. Since its initial deployment in 2013, the service has expanded from a handful of workloads to production roles spanning distributed filesystem metadata, event counting, and product data for app features. The combination of tunable durability, consistency, availability, and latency guarantees—all configurable per read or write—has made the service the default answer for most small key-value storage needs inside Facebook.

Transactions with serializable guarantees

ZippyDB offers transactions and conditional writes for workloads that require atomic read-modify-write operations across a set of keys. Every transaction is serializable at the shard level by default; lower isolation levels aren't supported. This choice keeps both the server-side implementation and client-side reasoning about concurrent transaction correctness straightforward.

Conflict detection relies on optimistic concurrency control. Clients typically read from a secondary to obtain a DB snapshot, assemble their write set, and send both the read and write sets to the primary for commitment. The primary determines whether any conflicting writes from other admitted transactions have occurred since the snapshot was taken. If no conflicts exist, the transaction is admitted and guaranteed to complete barring server failures. The primary tracks recent writes from admitted transactions within the same epoch to resolve conflicts; transactions that would span multiple epochs are rejected, avoiding the need to replicate write-set tracking state. The write history on the primary is periodically purged to control memory usage, and because a complete history isn't kept, the primary maintains a minimum tracked version to reject reads against older snapshots, preserving serializability. Read-only transactions follow the same path, simply with an empty write set.

Conditional writes are implemented as server-side transactions, offering a more ergonomic client API for cases where a client wants an atomic update guided by shared preconditions such as key_present, key_not_present, and value_matches_or_key_not_present. On receipt, the primary sets up a transaction context and translates the preconditions and writes into a standard transaction, reusing all existing transaction machinery. This API can outperform regular transactions whenever preconditions can be evaluated without performing a read first.

Evolution and roadmap

General-purpose distributed key-value stores are recurring building blocks for product backends and for storing metadata in infrastructure services. A scalable, strongly consistent, fault-tolerant store requires balancing many system trade-offs to serve varied real-world workloads. ZippyDB has been running in production for over six years and has seen steep adoption, thanks to its flexibility in tuning efficiency, availability, and performance per use case. As a shared service, it also pools capacity and lets teams avoid building and operating their own storage systems.

ZippyDB isn't done evolving. Active work includes architectural shifts toward storage-compute disaggregation, revised membership management, faster failure detection and recovery, and changes to distributed transaction handling, all aimed at adapting to shifting ecosystem and product needs.