Why Delta exists

Meta runs many storage systems, each tuned to different workloads and consistency requirements. Over time the company has worked to consolidate those systems, but certain workloads benefit from a purpose-built store. One such workload is the storage of build and distribution artifacts — packages that need to survive infrastructure failures and support disaster recovery and bootstrap scenarios.

That realization led to Delta, an object storage service designed from the ground up for reliability and recoverability rather than performance or efficiency. Delta sits at the very bottom of Meta’s infrastructure stack, providing the primitive on which the rest of the infrastructure’s availability depends. For bootstrap systems, any added complexity must make the solution more reliable; performance and efficiency are secondary concerns. The data stored in Delta must also be backed up so that recovery remains possible even when disaster strikes.

What Delta is (and isn’t)

Delta is a simple, reliable, scalable object storage service with minimal dependencies. It exposes just four operations: put, get, delete, and list. The service trades away latency and storage efficiency to gain simplicity and reliability. It is horizontally scalable, and where soft dependencies exist, it has appropriate failover strategies in place.

Delta is not a general-purpose storage system — its core tenets are resiliency and data protection for low-dependency systems. It is not a filesystem, so it does not expose POSIX or other filesystem semantics. And it is not optimized for storage efficiency, latency, or throughput.

Chain replication at the core

Delta productionizes chain replication, a strategy for coordinating clusters of fail-stop storage servers that provides high throughput and availability along with strong consistency guarantees.

In chain replication, servers are arranged linearly, like a linked list. Each chain contains a sequence of servers holding redundant copies of objects. The first server in the chain is the head; the last is the tail.

All writes go to the head, and updates pipeline down the chain from server to server. Once every server has persisted the update, the tail responds to the client. Reads are served only by the tail, which guarantees that whatever a client reads has already been replicated across the full chain — strong consistency without complex coordination.

Chain replication vs. quorum replication

Chain replication is not the most storage-efficient option available. It stores full redundant copies of the entire data set on every host in each chain, whereas erasure coding can replicate data fragments more efficiently. It also has higher write latency than quorum-based approaches, because writes complete only after every host in the chain has acknowledged, not just a write quorum.

But chain replication has compensating strengths:

  • Fault tolerance: A chain of n nodes can tolerate up to n – 2 failures without losing availability. Quorum systems need at least w hosts to serve writes and r hosts to serve reads, where w and r are the write and read quorum sizes.
  • Performance: In primary-backup replication, reads can be served by any backup. In classic chain replication, only the tail serves reads, though Delta has optimized this (described later).
  • Simpler consensus: Quorum systems require complex consensus and leader election. Chain replication narrows the problem to a simple chain-host mapping — the head is always the write leader, with no election needed.

Anatomy of a Delta bucket

A Delta bucket is composed of multiple chains, each typically containing four or more servers (the count can vary based on the desired replication factor). Each chain acts as a replica set serving a slice of the data and traffic, effectively a logical shard of the client data set. Servers within a chain are distributed across different failure domains — power, network, and so on — so data remains durable and available even if entire failure domains go down.

The bucket config holds the authoritative chain-host mapping for the bucket layout. That config is updated as servers and chains are added or removed.

When a client accesses an object in a bucket, a consistent hash of the object name selects the appropriate chain. Writes are always directed to the chain head, which writes to local storage and forwards the write to the next host in the chain. The client gets an acknowledgment only after the last host durably stores the data. Reads go to the tail, ensuring fully replicated data is all a client can see.

Delta scales horizontally by adding servers to a bucket and rebalancing chains without impacting availability or throughput. One rebalancing tactic: servers that hold the most chains hand some off to newly added servers. Rebalancing still respects the failure-domain distribution requirements for the bucket layout.

Failure detection and recovery

Failures in a Delta deployment come from hosts going down, network partitions, planned maintenance, operational mishaps, or unexpected events. Delta assumes servers are fail-stop: each server halts on failure rather than making erroneous state transitions, and the halted state is detectable by the environment.

In a bucket of n chains, each with more than one host, sibling hosts sharing a chain detect a misbehaving or partitioned host and report it. Detection happens through simple heartbeats or failures while transmitting acknowledgments or requests up and down the chain. When multiple hosts suspect one target, that host is removed from all its chains and sent for repair, and the bucket config is updated.

Two settings govern how aggressively a host is suspected, and both involve trade-offs:

  • Timeout settings: Performance testing determined the right link timeout. Too short, and transient network issues cause flapping. Too long, and operation latency suffers — clients may even time out waiting for a response.
  • Suspicious host voting: The vote threshold for kicking a host out of its chains cannot be one, since two hosts in a chain could each vote the other out. But a very high threshold lets an unhealthy host linger and degrade service. Since each host belongs to multiple chains and communicates frequently with upstream and downstream peers, a voting limit of two works well in practice. Automated repair flows keep false positives manageable.

Once a faulty host recovers, it rejoins all the chains it served before being evicted. New hosts are always added at the tail end of a chain. On rejoining, a host synchronizes with updates that occurred while it was out of the chain: it scans objects on the upstream host and copies anything missing or outdated. During this reconstruction window, the host can still accept writes from upstream, but must defer reads until it is fully synchronized.

The same process handles both returning suspected hosts and adding new capacity to a bucket.

Scaling reads with apportioned queries

Chain replication's read path has inherent bottlenecks. Since the tail node is the only server serving reads, it becomes a hotspot, and the tail's throughput caps the entire chain's read capacity.

Delta addresses these limits with an approach inspired by chain replication with apportioned queries. In this model, every node in the chain can serve read requests. Before responding, a non-tail node performs a crucial validation: it checks with the tail whether its local copy of the object is clean, meaning it has been committed by all servers in the chain. If the node's copy is dirty — not yet fully replicated — the tail informs it of the latest clean version to return. This mechanism preserves the strong consistency guarantees of chain replication while distributing the read load.

The extra network call each non-tail node makes to the tail is relatively cheap compared to serving a full client read. The tradeoff pays off: read throughput and effective chain bandwidth scale roughly linearly with chain length, without adding significant latency for clients.

Automated chain repair

Hardware failures and network partitions are common enough in large clusters that manual intervention isn't practical. Delta relies on a control plane service (CPS) to automate fleet management. Each CPS instance monitors a set of Delta buckets and is responsible for repairing chains with missing links.

The CPS applies several principles when rebuilding chains:

  • Failure domain awareness: Repairs maintain the bucket's failure domain distribution so hosts are spread evenly across all domains.
  • Load balancing: The CPS ensures chains are distributed uniformly across servers to prevent any single host from becoming overloaded.
  • Host preference: When a chain is missing a link, the CPS first tries to repair with the original host rather than a new one. Resyncing partial chain contents to the original host is far less compute-intensive than a full sync to a fresh server.
  • Sanity checks: Before reintroducing a host into a chain, the CPS runs detailed checks to confirm the host is healthy.
  • Standby pool: The CPS maintains a pool of healthy standby servers. If a chain loses more than 50% of its hosts, a standby is added immediately to prevent severe underhosting from threatening bucket availability, while still applying the other principles on a best-effort basis.

Global replication

In Delta's original design, clients needing multi-region blob storage had to send requests to each region themselves. That forced users to track object locations and manage redundancy preferences manually. The goal is for a client to issue a single put request and let the storage fabric propagate the data, and likewise for reads to be served from the most optimal available source.

Delta now supports global replication using a hybrid model: blobs are replicated synchronously to a few regions and asynchronously to the rest. This arrangement keeps client latency low while ensuring other regions achieve eventual consistency. If a region suffers a network partition or outage, the system automatically excludes it from geo-replication until it recovers, at which point the region is asynchronously backfilled with missing blobs.

Disaster recovery

A core design tenet for Delta is to keep dependencies minimal while maintaining a solid disaster recovery story. To that end, Delta integrates with archival services that continuously back up client blobs to cold storage, and it can restore objects from those archives in a continuous manner. This out-of-box integration provides reliable recoverability even in severely degraded environments, and it has led several partner teams to adopt Delta specifically for these guarantees.

Future direction

Delta's near-term focus is a centralized backup and restore service for core infrastructure systems at Meta. Any reliable stateful service should be able to snapshot its internal state for backups and rehydrate from a snapshot when restoring. Delta aims to serve as a gateway to all archival services, providing a unified backup offering while continuing to strengthen the reliability of its disaster preparedness and recovery capabilities.