At Meta, inference fleets can be globally constrained while meaningful pockets of capacity sit locally idle. The company built a system that aggregates those pockets into a shared opportunistic capacity pool and uses them to improve ads recommendations, without ordering additional capacity. The work is optional by construction: when opportunistic capacity disappears, the system gives up incremental ranking work rather than degrading baseline serving.

Consuming this capacity safely is the hard part. It is spread across team-owned allocations, regions and accelerator types, and it can be reclaimed with little notice. The ads inference system therefore has to adapt its workload as supply changes while keeping the capacity reserved for baseline traffic within its latency and reliability targets.

Why idle capacity accumulates

Inference fleets are provisioned for peak demand under strict latency and reliability targets, but demand and capacity do not align evenly across time, regions, resource combinations or accelerator types. Several mismatches are common:

  • Peak provisioning versus demand variation: fleets are sized for peak traffic, so capacity needed at peak can sit idle at other times of day or season.
  • Model development versus capacity delivery: capacity arrives on supply-chain timelines while models are developed and launched on separate schedules, leaving a gap between availability and consumption.
  • Regional resource ratios: accelerator and supporting resources such as CPUs may not be available in the required proportions, so one resource stays idle while another limits serving.
  • Accelerator types: fleets span vendors, generations and architectures, and workloads are not always portable across them, stranding capacity that cannot serve the available demand.

The result is a fleet that is globally constrained while local pockets of capacity go unused. At Meta's scale those temporary pockets add up to substantial capacity, and using them safely produces better ads recommendations.

What the workload has to look like

Because opportunistic capacity is preemptible, volatile and carries no long-term guarantees, not every workload can absorb it. Three structural properties matter:

  • Elasticity: the workload can add or remove a large amount of optional work quickly without changing baseline expectations.
  • Sustained quality at scale: output quality keeps improving across the range the pool can supply — a property that often means the workload already consumes a lot of dedicated capacity.
  • Few cross-service dependencies: a self-contained workload limits how many upstream and downstream systems must react to volatile capacity.

Two complementary levers meet these criteria.

Rank More Ads and Serve Better Models

Ads recommendation selects a small number of ads from millions of candidates in under a second, using a sequence of ranking stages in which increasingly powerful models evaluate progressively smaller candidate sets. Rank More Ads (RMA) raises the number of candidates the ranking stages evaluate whenever opportunistic capacity is available; more candidates improve the chance of finding ads relevant to the user. Because additional candidates pass through CPU-heavy upstream stages such as retrieval and feature processing as well as accelerator-heavy inference, RMA generally needs surplus in both CPUs and accelerators.

Serve Better Models (SBM) instead adjusts the share of requests evaluated by more powerful models as accelerator capacity changes. More powerful models improve prediction quality but are not efficient to apply to every request on dedicated capacity. SBM holds the number of ads entering the ranking stages constant, so upstream CPU demand stays comparatively stable, letting it consume accelerator-only surplus that RMA cannot. The SBM model router uses request-value signals to prioritize the requests expected to benefit most.

The two levers draw on different resource combinations, so together they cover more of the opportunistic pool than either would alone. Their incremental impact also shifts with traffic, current coverage and the available resource mix: at any moment the next unit of capacity may be worth more as additional ranked candidates or as extra requests routed to a powerful model for prediction accuracy. Meta periodically rebalances the allocation between RMA and SBM based on measured performance and resource requirements.

Keeping baseline traffic out of harm's way

Opportunistic capacity can be reclaimed faster than the control loops that regulate work across a serving fleet of this size. The system is therefore designed around a single requirement: losing opportunistic capacity may reduce incremental ranking work, but it must never overload the capacity serving the baseline workload. Isolation, per-workload feedback loops and fast local shedding enforce that.

Physical isolation for SBM, logical isolation for RMA

SBM isolates baseline serving physically. Baseline and more powerful models have separate routing targets: dedicated replicas are sized to serve requests on the baseline model, while replicas on opportunistic capacity serve the requests selected for the more powerful model. When opportunistic capacity contracts, fewer requests are routed to the powerful model and the rest fall back to the baseline model, whose capacity is already provisioned for them.

RMA's isolation is logical. The system computes how much additional candidate-ranking work the model replicas on opportunistic capacity can support and sets a global RMA budget from total capacity. Individual replicas may carry a mix of baseline and additional work, and requests do not need physical separation; baseline traffic is protected by capping total incremental work at what those replicas can absorb. In a contraction, RMA lowers its global candidate-volume target and SBM lowers its powerful-model coverage — in both cases by discarding optional work, never by spilling it onto reserved baseline capacity.

A PID loop sizes candidate volume

RMA runs a closed loop combining a central proportional-integral-derivative (PID) controller with client-side enforcement. Simplified, the controller reads observed tail latency and derives a target for the number of ads sent to downstream ranking, and clients apply that target to new requests. Latency below target raises the number of ads ranked; rising latency lowers it. RMA thus converges on a sustainable candidate volume as the capacity situation changes.

Value-aware model routing

SBM has a matching loop, with central control of target coverage and client-side routing. The client model router groups requests into segments by a request-value signal, meaning the predicted likelihood of user engagement with the ad, and an explore-and-exploit process periodically measures the incremental outcome of the powerful model in each segment and updates the split between models. Absolute request value alone is not a reliable guide to incremental benefit, and continued exploratory traffic across segments keeps fresh observations coming instead of entrenching old routing decisions. A calibration loop holds the realized traffic share near target as the traffic mix shifts across regions and over time. Under contraction, the controller cuts target coverage and the client sends fewer requests down the powerful-model path; the rest continue on the baseline model.

Shedding inside milliseconds

The central components of the RMA and SBM loops operate on the order of minutes, so a sudden preemption can strip capacity before clients see new targets. Two value-aware mechanisms react immediately:

  • Server-side shedding: when the pool loses capacity before clients catch up, servers drop lower-value opportunistic work on the spot — RMA reduces candidates processed within individual requests, and SBM applies an equivalent value-aware policy to requests routed to the powerful model. This absorbs rapid change close to the serving bottleneck.
  • Client-side shedding: RMA and SBM clients treat the latest controller targets as a guideline but also watch server status directly, and shed lower-value opportunistic work when they observe overload — RMA requests fewer additional ads, SBM routes fewer requests to the powerful model.

Both react within a few milliseconds, covering the interval until the controller-client loops converge. Both use value signals to choose what to remove, which limits the effect of the transition.

Where the Work Goes Next

Extending elasticity within RMA and SBM

The immediate goal is to widen the operating range of both recommenders and to add elasticity levers that go beyond them. For RMA, that means handling higher candidate volumes at each stage of ranking; for SBM, it means adopting more powerful models whose predictions improve as more accelerator capacity is made available. Precomputation elasticity is being developed as a further lever, since work performed ahead of request time is less tied to where and when live traffic lands.

Making more of the pool reachable

Opportunistic capacity is scattered across regions and accelerator types, so it only pays off where a compatible model and its supporting resources coexist. Model portability and cross-region inference are the two directions under development to raise the share of that capacity that can be put to work.

Letting an optimizer decide

Which lever deserves the next unit of capacity depends on traffic, coverage and the resource mix at that moment: it may be better spent ranking extra candidates, or better spent routing more requests to a strong model. An optimization-based system is being built to set that allocation automatically according to user and advertiser benefit, maximizing return on the capacity subject to constraints including regional restrictions, hardware compatibility, and how long the idle capacity must remain durable for each use case.

Contributors

Cross-functional contributors across Meta: David Amsallem, Mohit Bharti, Maddie Digne, Stephen Dimmick, Maomao Ding, Amit Dutta, Sebin George, Aashik Gowda, Atul Goyal, Mohak Goyal, Nelson Guo, Tiger Hao, Yaning Huang, Dianshi Li, Yang Li, Bai Liu, Bofeng Liu, Wei Liu, Yisi Liu, Daniel Molina Hurtado, Maciej Naruszewicz, Arka Pal, Annan Pan, Brian Park, Brian Peschka, Michael Shao, Bikash Sharma, Alexander Sun, Weiman Sun, Kaiyue Tao, Patrick Taylor, Kevin Tong, Ellie Wen, Chao Xie, Lu Yang, Sihan You, Hansen Zhang, and Anthony Zhu.

Sponsorship and support: Neeraj Agrawal, Justin Charlap, Brian Ellis, Prabhakar Goyal, Hong Jiang, Shri Karandikar, Santanu Kolay, Uladzimir Pashkevich, Rob Schwartzberg, Eric Shan, Matt Steiner, Nathan Yan, Tak Yan, Caijie Zhang, and Alex Zhong.