Viewing the world as a computer: Global capacity management

From regional to global: why capacity management had to change

Meta’s data center footprint has grown to 14 regions worldwide, and that growth is putting pressure on the abstractions that previously served the company well. Systems like Twine for cluster management and RAS for region-wide resource allocation let service owners treat machines as interchangeable within a single region. But those regional tools don’t extend to the harder question of where a service should live in the first place.

Today that decision often falls on service owners, who hard-code regions and manually calculate what capacity they need in each one to stay disaster-ready. The practical result: infrastructure that can't easily shift work between regions as hardware gets refreshed, new regions come online, or goals change. Meta wants service owners to become region-agnostic, not just machine-agnostic, and that requires a new layer of global capacity management.

Two new systems aim to provide it: Global Reservations Service, which handles capacity placement for latency-tolerant services, and Regional Fluidity, which does the same for latency-sensitive ones. Both rest on a common premise—the infrastructure should know the service owner's intent, not the specific regions where capacity should sit, and figure out the details automatically.

Two service classes, two placement problems

Meta scoped its global capacity work to two categories of service. Latency-sensitive, stateless services power interactions where fast response matters—viewing photos in the Facebook or Instagram mobile apps. Latency-tolerant services can withstand some delay—think of uploading a large file or catching up on a friend's comments on a video.

Latency-sensitive services carry a complication: their placement can't be optimized in isolation. Their upstream and downstream dependencies matter too, because all of them need to be near one another to keep latency low. Latency-tolerant services have no such constraint, which makes their global placement problem much more tractable.

Global reservations: making fault tolerance declarative

Global Reservations Service sits on top of RAS, which already provides regional capacity guarantees. Under the old model, a service owner had to reason about how much capacity to reserve in each region to be protected against a single-region failure. As the number of regions grew, that reasoning multiplied in complexity.

The shift is from many regional reservations to one global request. A service owner declares, once, how much capacity the service needs and roughly where it's acceptable to run; the Global Reservations Service then decides how to split that request across specific regions. Consider a hypothetical request for 300 servers somewhere in continental North America:

The service in this example is indifferent to which specific region provides the machines, as long as it can survive the loss of any one region. The Global Reservations Service might place 150 servers in each of three regions, 450 total. Losing any single region leaves 300 servers running—exactly what the service requested. As regions multiply, the buffer becomes more efficient; instead of 150 extra servers in three regions, the same 300-server request can be satisfied with 30 servers scattered across 11 regions, for 330 total, since the loss of any one region is a smaller share of the total.

The solver treats placement as an assignment problem

At the core of Global Reservations Service is a solver that assigns machines—the objects—to reservation bins. That assignment is encoded as binary variables, where a 1 indicates that a given machine is allocated to a given global reservation. Since a machine can only belong to one reservation, the variables down a machine's row must sum to 1, with a special “unassigned” bin catching any servers that aren't reserved.

A reservation's column must sum to at least the amount the service owner requested. The fault-tolerance constraint adds another condition: the number of machines assigned to a reservation, minus the count in its largest region, must still meet the request—so the remaining regions can absorb a single-region loss.

The solver dispatches these constraints and objectives as a mixed-integer linear programming problem. For example, minimizing total allocated servers across the fleet is just an objective that can be expressed directly:

Once assignments are made, the solver generates the regional reservations that RAS will manage, and it revisits the assignment continuously to shift placement toward whatever objective is current. A hardware refresh that removes capacity from one region doesn't have to become a service-owner problem—the global solver can reallocate across remaining regions automatically, as long as service owners have expressed their requirements as intent rather than as a fixed topology.

Why this stops short of latency-sensitive services

Global reservations are a clean answer for services that tolerate cross-region movement. But they don't address the tighter coupling of latency-sensitive services, where the system must consider the placement of not just the service but its entire dependency chain. That calls for a different mechanism, which Regional Fluidity is meant to provide.

Regional Fluidity for latency-sensitive workloads

Latency-sensitive services require a different capacity model. Such services typically need placement constraints that avoid or reduce cross-region network calls, so capacity cannot be sprinkled arbitrarily across regions. Instead, capacity must be allocated proportionally to where the incoming demand originates. This means we need to attribute service demand back to its sources: for instance, requests originating from the Facebook app versus the Instagram app.

Attribution is achieved through a distributed tracing framework. If a request from one demand source calls a chain of services, tracing tells us which demand source drove each call. Two properties make this model practical:

• Each demand source can be shifted independently through global traffic load balancers, so we can move Facebook traffic without altering Instagram routing.
• Once a demand-source request enters a region, it should complete there or be handed off to another independently shiftable demand source. Transitively called services must therefore stay within the region.

Rebalancing by shifting demand sources

With demand sources identified, capacity distribution across regions becomes an exercise in redistributing demand. To reclaim capacity in one region and use a surplus in another, capacity is first provisioned in the destination region, then a portion of a demand source is shifted over, and finally the now-idle capacity in the source region is reclaimed. The order matters: adding capacity before shifting traffic prevents overload.

In practice, even a simple two-region shift involves multiple steps. Scaled to many regions, the problem becomes a bin-packing assignment: given nodes and services across regions, how much capacity of each service can go where?

Planning and executing global shifts

The planning layer, called the Solver, treats service dependencies and traffic ratios as constraints. It incorporates demand attribution and available supply to produce a feasible placement plan across demand sources, then applies global optimizations across regions.

Regional Fluidity thus adds three components to the capacity-management toolchain:

• Modeling that captures each service's latency and geographic distribution requirements
• Solvers that handle the global placement assignment with constraint honoring and global objectives
• Automation that rebalances services across regions and redistributes the driving demand

The Orchestrator executes the plan end to end, sequencing actions to keep every region in a safe state throughout.

A complete regional shift of a demand source involves the following sequence:

1. Add additional capacity in the destination region
2. Add replicas of the relevant Twine jobs in that region
3. Increase traffic in the destination region while decreasing it in the source region
4. Remove replicas of the Twine jobs in the source region
5. Reclaim capacity in the source region

The temporary over-provisioning required by this sequence is absorbed by enlarging the double-occupancy buffer at marginally higher cost, enabling safe automatic shifts.

This capability yields three concrete benefits:

• Homogenization of the hardware footprint toward standard region types. With services that can move, hardware becomes more fungible, reducing stranded power from mismatches between workload needs and regional infrastructure.
• Safe decoupling of service capacity management from regional supply planning. Service owners no longer reason about regional distribution manually.
• Global optimization that can trade regional inefficiencies for better overall outcomes, such as improving resource utilization at a global scale.

The road to a globally managed infrastructure

The mechanisms described so far sit at two ends of a spectrum: latency-tolerant reservation-based services and latency-sensitive placement-constrained services. Much of the middle remains unaddressed. AI training workloads, for instance, are more fluid than storage but need locality to training data. Stateful and multitenant systems like databases and caches present yet another set of constraints, with data-copy costs that lengthen migration times.

Bringing these workloads under global capacity management starts by changing service owners' mental models, shifting the contract from regional Twine jobs and reservations to global abstractions with explicit intent. A continuously running capacity regionalizer determines the exact regional capacity breakdown and safely orchestrates the required changes. Services needing more deliberate planning or orchestration are handled by the capacity orchestrator.

Globally sharded systems offer an early preview of how such orchestration might extend. For a regional shift, the orchestrator could insert extra sequenced steps: after the Twine job is upsized in the destination region but before traffic is moved, it instructs the shard placement system to build new shard replicas there. After the shift, it tells the same system to drop replicas in the old region before the Twine job is downsized. Such sequencing makes regional shifts safer and more automated for sharded services.

Much of this remains in progress, with open questions driving the roadmap:

• Which global abstractions best capture service intent while leaving the infrastructure latitude to improve placement autonomously?
• How do we safely orchestrate regional shifts for every service type across the spectrum?
• How do we model stateful, multitenant services as the set of regions and failure characteristics grows, and handle automatic shifts safely?
• How do we evolve disaster-readiness practices without adding burden to service owners?

Global capacity management marks the first step toward treating the entire infrastructure as one pool of resources. It is an early step, but the trajectory is set: service owners reason globally, infrastructure handles regional placement, and capacity flows to wherever it is needed. Up next is closing the gap for the many service types that today still require a human in the loop.