Scaling services with Shard Manager

Why sharded services need a manager

Scaling a stateless web tier is straightforward: any server can handle any request, so traffic can be spread with simple strategies like round robin. Stateful tiers are a different problem. Data must be deterministically distributed across servers, and the distribution scheme must handle growth without excessive data movement.

A simple modulo-based hash, hash(data_key) % num_servers, spreads data but reshuffles most of it whenever the server set changes. Consistent hashing limits reshuffling to a small subset, but it requires fine-grained keys to balance load statistically and cannot express constraint-based placement, such as keeping EU user data in European data centers. Distributed caches can live with these limitations; most other stateful systems cannot.

Explicit sharding is the more general alternative. Data is partitioned into discrete shards, each allocated to a server. Each shard can have multiple replicas with distinct roles, such as primary and secondary, depending on consistency requirements. Because shard-to-server allocation is an explicit computation, it can incorporate constraints like locality that hashing cannot express.

Explicit sharding is flexible, but it requires operational machinery that many teams end up building by hand:

• Failover — diverting traffic from failed servers and rebuilding replicas on healthy ones, including proactive moves before planned maintenance.
• Load balancing — adjusting which shards each server hosts so utilization stays even as workloads shift.
• Shard scaling — changing replication factor per shard so per-replica load stays optimal as client traffic fluctuates.

At Facebook, service teams were independently building custom solutions to these problems, often with uneven coverage — failover was common, real load balancing less so. The result was inconsistent reliability and high operational overhead. Shard Manager was built as a generic platform to fill that gap.

Shard Manager as a platform

Shard Manager has been adopted by hundreds of sharded applications, managing tens of millions of shard replicas across hundreds of thousands of servers. These applications support user-facing products including the Facebook app, Messenger, WhatsApp, and Instagram.

Adoption is broad in complexity as well as count. Applications range from simple counter services running on a few dozen servers to Paxos-based global storage running on tens of thousands. Example use cases span the spectrum, as shown below.

Several design decisions drove this adoption:

• A minimal integration surface — applications implement a small interface with add_shard and drop_shard primitives.
• Intent-based specification — applications declare their reliability and efficiency requirements rather than implementing mechanics.
• A generic constrained optimization solver — this provides flexible load balancing and makes it easy to add new balancing strategies.
• Deep infrastructure integration — capacity and container management are built in, delivering an end-to-end solution that goes beyond what platforms like Apache Helix offer, including support for Paxos-based storage systems.

Three application types

Shard Manager abstracts what sharded applications have in common into three categories: primary only, secondaries only, and primary-secondaries.

Primary only

Each shard has a single primary replica. These applications typically store state externally and use a shard as a worker that claims its own data range, processes it, and writes results back — with Shard Manager guaranteeing at most one primary per shard to prevent duplicate processing. Stream processing is a representative case. This is conceptually similar to ZooKeeper lock-based coordination.

Secondaries only

Each shard has multiple replicas of equal role. Replication provides fault tolerance, and the replication factor can be adjusted per shard so hot shards get more copies to spread read load. These applications are typically read-only without strong consistency requirements: they load data or models from external storage, cache locally, and serve reads. Machine learning inference platforms fit this model.

Primary-secondaries

Each shard has one primary and at least one secondary. The primary accepts writes and drives replication; secondaries provide redundancy and can serve reads. These are storage systems with strict consistency and durability requirements, implemented with protocols like Paxos — Zippy DB is one example.

As of August 2020, primary-only applications made up 67 percent of Shard Manager applications due to their architectural simplicity, but accounted for only 17 percent of servers — the other two types tend to be much larger in deployment size.

Building on Shard Manager

Once the workload is sliced into shards and the application type is chosen, building on Shard Manager follows a consistent pattern regardless of use case:

1. Link the Shard Manager library and implement the shard state transition interface, plugging in the business logic.
2. Provide an intent-based specification describing constraints. Out-of-the-box functionality covers fault tolerance, load balancing, shard scaling, and operational safety.
3. Use the common routing library on the client side for shard-specific requests.

The state transition interface

The core interface is deliberately small, built on these primitives:

status add_shard(shard_id)
status drop_shard(shard_id)

add_shard instructs a server to load a shard by ID; the return value indicates progress or errors. drop_shard instructs a server to drop a shard and stop serving requests for it. What loading actually means is application-specific: a storage service uses add_shard to trigger data transfer from a peer replica, while a machine learning platform uses it to load a model from remote storage.

On top of these primitives, Shard Manager runs a shard move protocol. To move a shard from an overloaded host A to a lightly loaded host B, it first calls drop_shard on A and waits for success, then calls add_shard on B. This guarantees at most one primary ever exists per shard.

The move protocol is safe, but it creates a window of unavailability while the shard is orphaned — unacceptable for user-facing applications. For those cases Shard Manager supports a seamless handoff protocol. Primary-secondaries applications also get two more primitives:

status change_role(shard_id, primary <-> secondary)
status update_membership(shard_id, [m1, m2, ...])

• change_role transitions a replica between secondary and primary.
• update_membership instructs a shard's primary to validate and execute replica membership changes, which matters for Paxos-based systems where correctness depends on carefully ordered membership updates.

These additional interfaces cover the advanced patterns that emerged from work with existing sharded applications, in addition to the two basic primitives that suffice for the majority of use cases.

Building for failure as the default state

Distributed systems treat failure as an expected condition, and Shard Manager's fault-tolerance features are built around that assumption. Replication factors are configurable per shard, with support for spreading replicas across definable fault domains — buildings for regional applications, regions for global ones — so a single domain failure cannot take down redundant copies.

Failure detection and failover are automatic but tunable. Applications can adjust failure detection latency and failover delay to balance the cost of rebuilding replicas against acceptable downtime. For network partitions, the trade-off between availability and consistency is left to the application. Failover throttling caps the rate of shard recovery so that a large outage does not cascade into overload on the surviving servers.

Continuous load balancing

Shard placement is reevaluated on an ongoing basis rather than fixed at deployment time. The balancing algorithm uses fine-grained per-server and per-shard data, which means it handles heterogeneous hardware generations and unevenly sharded workloads. Load and capacity metrics are collected periodically from applications, allowing the system to react to changes in usage or to capacity that depends on dynamic resources such as available disk.

Balancing is multi-resource: compute, memory, and storage are simultaneously optimized with user-configured priorities, keeping bottleneck resources within acceptable bounds while distributing less critical ones as evenly as possible. Move throttling, both globally and per server, prevents balancing activity from disrupting healthy servers. This handles both spatial variability — differences across servers and shards — and temporal variability, where a shard's load profile shifts over time.

Elastic shard scaling

Many Facebook applications serve user traffic that follows a diurnal pattern with substantial peak-to-off-peak variation. Shard Manager supports elastic scaling by adjusting the replication factor dynamically: when average per-replica load for a shard leaves a user-configured acceptable range, replicas are added or dropped to bring it back. Throttles limit how many replicas can be added or removed within a given period.

Operational events as first-class concerns

Operational activity — binary updates, hardware repair, kernel upgrades — is handled explicitly to minimize reliability impact. Shard Manager is co-designed with the Twine container management system; Twine aggregates operational events, converts them into container life-cycle changes like stop, restart, or move, and communicates them to the Shard Manager Scheduler through the TaskControl interface.

The scheduler evaluates each event's disruptiveness and duration, then makes proactive shard moves to protect availability. A core invariant ensures every shard retains at least one healthy replica; for Paxos-based applications requiring a majority quorum, a variant guarantees a healthy majority. Applications tune the trade-off between operational safety and efficiency, for example by setting a limit on how many shards may be affected simultaneously.

Client-side routing

Request routing relies on a common library used across Facebook. Given an application name and shard ID, it returns an RPC client object; the details of locating the servers that host the shard are handled inside the create_rpc_client call. This keeps shard discovery transparent to the calling code.

rpc_client = create_rpc_client(app_name, shard_id)
rpc_client.foo(...)

Infrastructure layering and division of responsibility

Facebook's infrastructure stack is organized in layers, each with clear scope:

1. Host management: The Resource Allowance System manages physical servers and allocates capacity to organizations.
2. Container management: Twine takes that capacity and allocates it to applications in containers.
3. Shard management: Shard Manager allocates shards within Twine-provided containers.
4. Sharded applications: Each shard runs the application's associated workload.
5. Products: User-facing applications consume the sharded back-end services.

Each layer depends functionally on the layer below it, but the stack is co-designed so that signals and events propagate upward. For Shard Manager, TaskControl provides the mechanism for this collaborative scheduling.

Design principles

Central control plane. Shard Manager is purely a control plane: it monitors state and orchestrates data movement across servers, with no application traffic in the data path. A central global view supports optimal allocation decisions and holistic coordination of planned events. If the control plane is unavailable, applications continue serving in degraded mode with their existing allocation.

Opaque shards with defined state transitions. Shards carry no meaning to Shard Manager; they can represent database instances, log groups, buckets of data, or any other entity. Applications implement a shard state transition interface, which keeps the platform general across very different use cases.

Chosen granularity. The system deliberately allocates hundreds of shards per server. Coarser granularity would degrade load-balancing quality; finer would raise overhead on the underlying infrastructure.

Constrained optimization. New allocation requirements are formulated as constraints fed into a generic constrained optimization solver, avoiding per-feature code complexity.

System architecture and data flow

Application owners supply a specification containing all required management information. The Shard Manager Scheduler is the central orchestration service: it tracks application state, detects changes such as server joins, failures, or load shifts, adjusts allocations, and drives state transitions through RPC calls to application servers.

On the application side, the Shard Manager library provides server membership and liveness checks, abstracted through ZooKeeper. Applications implement the state transition interface and may expose dynamic load metrics, which the scheduler collects. The scheduler publishes the current shard allocation into a scalable service discovery system, and clients learn endpoints from it.

No client request passes through the scheduler: after endpoint discovery, clients talk directly to application servers.

Open problems and the road ahead

Shard Manager has been in production for nine years, but the design targets several areas of future growth. Three challenges stand out:

1. Reaching tens of millions of shards per application, by internally partitioning large applications into smaller independent parts.
2. Supporting more complex applications through greater modularity, letting users customize behavior while the core stays simple.
3. Simplifying onboarding for small applications where the current abstraction is too heavyweight.

The platform continues to aim for a generic sharding solution across Facebook's infrastructure, adapting as scale and application diversity grow.