Keeping Meta’s GPU fleet healthy under constant change

Meta’s data centers now support one of the largest AI training infrastructures in the world, spanning dozens of GPU clusters that are planned to scale to 600,000 GPUs within the next year. Those clusters run thousands of training jobs daily for hundreds of internal teams. Job sizes vary wildly — from a single GPU running for a couple of minutes to generative models with trillions of parameters spanning thousands of synchronized hosts. The latter are highly sensitive to interruption, and tightly bound to specific hardware, backend networks, and topologies, so the software-to-hardware dependency chain is deep and brittle.

Keeping that fleet operational without sacrificing capacity availability required rethinking maintenance. The core challenge: how do you update and verify software and firmware components across the networking path and host stack when you cannot take large chunks of capacity down, and when any single bad host can degrade an entire multi-thousand-host training run?

Why GPU training changes the maintenance calculus

Traditional capacity maintenance assumes some tolerance for downtime and host variance. AI training pushes back on both. Several characteristics drive the approach:

  • Capacity guarantees. Many training jobs are time-critical, recurring, or online. Large capacity blocks cannot be taken out of production by default.
  • Bad hosts are disproportionately harmful. Because training requires all hosts to stay synchronized, a single slower or subtly broken host — a non-fatal hardware issue, a networking glitch — can stall the entire job.
  • Low interruption tolerance. Distributed jobs share state across hosts; interruptions are expensive and disruptive.
  • Rollout caution. The AI software stack is deep, and failures are hard to localize. New components need careful introduction.
  • Host consistency matters. Cross-host jobs rarely hit hard incompatibility beyond the CUDA version, but cluster consistency is critical for debugging and avoiding major incidents.

Meta’s environment adds further constraints. The hardware is bespoke, using the newest chips and speed-optimized backend networks. The software stack is kept intentionally current so firmware upgrades can unlock new features or reduce failure rates. That means more than 30 distinct maintenance operations, updates across 50 different components, three separate host-verification tasks, and thousands of disruptive AI host tasks every single day — all executed while preserving capacity guarantees for workloads ranging from single-host offline tasks to 24/7 online training jobs.

An overview of different maintenance rollouts happening on Meta capacity over time with overlapping durations.

Sliding upgrades instead of big-bang maintenance

Smaller environments can keep clusters in a uniform state and upgrade the entire cluster — all firmware and software — within a single maintenance window. At Meta’s scale, that approach is operationally infeasible and risky. With many overlapping in-flight changes at any time, including consistently applied verification tasks, Meta instead ensures components remain compatible and rolls upgrades up in a sliding fashion. This preserves capacity availability while allowing the fleet to stay current.

In practice, maintenance is organized into domains that isolate a small percentage of capacity for downtime at any moment. This gives the fleet predictable capacity: everything except one maintenance domain remains up 24/7, which is mandatory for online and recurring training. Longer-running upgrades may have relaxed rollout guarantees and can be scheduled across multiple maintenance cycles, so multiple upgrades can overlap — and, when beneficial, be aligned.

For AI capacity specifically, Meta designed optimized domains to fit different kinds of AI workloads, strict service-level objectives, and contracts that let services avoid maintenance-train interruptions whenever possible.

Gradual rollouts with a consistent AI job stack

An illustration of the distinction between higher-level components of the AI stack—such as the CUDA drivers, and the lower-level components involved in the training job,.

At scale, all disruptive rollouts outside special cases are gradual. That means different servers in a cluster can briefly run different host stacks — a normal pattern in traditional capacity, but difficult in AI training, where jobs are closely tied to hardware. Meta’s solution separates the layers: the AI job itself, including the CUDA library, stays consistent across hosts, but lower-level components roll out gradually. The distinction is practical. Lower-level components can take hours to install, configure, or require a host reboot, while the higher-level components inside the job container can restart fluidly.

That simple-sounding split required substantial engineering: careful testing at all lower levels, special monitoring, and tight collaboration with vendors. The AI stack overall has matured considerably over the past three years, and Meta added tooling for the rare upgrades that break compatibility.

Maintenance domain sizing

Maintenance domains are selected based on the amount of buffer-reserved capacity (the smaller the better) and the amount of interruptions we cause to training jobs (the bigger the better).

The size of a maintenance domain — the percentage of capacity taken down at once — was a key optimization. Working directly with AI teams, Meta tailored that size to balance two costs: the cost of interrupting AI jobs and the capacity lost during maintenance. Since AI interruption costs run high, optimizing that tradeoff significantly reduced the maintenance overhead for AI capacity.

OpsPlanner: coordinating disruptive work

Consistency requirements add an orchestration challenge. Moving to a new CUDA version, for instance, may require all capacity on a new driver version first. That is difficult across thousands of hosts with many planned and unplanned operations that may overlap. Meta unified this work in OpsPlanner, its disruptive-work orchestrator. OpsPlanner handles overlapping scopes of operations, serializes them correctly, and safely moves hosts in and out of production. Its built-in handover flow ensures escalation behavior is correct and prevents overlaps and deadlocks. Upgrades are guaranteed to be applied before hosts return to production, and the tool manages planned maintenance and failure buffers with their own safeguards. It currently processes a million operations per day.

Example scenarios illustrating the disruptive work scheduler the OpsPlanner needs to handle to ensure host consistency.

The safety stack underneath includes automatic stop of maintenance trains if maintenance or failure buffers are exhausted, automatic offboarding of failing upgrades, and phased rollouts so only well-tested changes reach global systems. For incidents that do require fast reaction, Meta can fall back on emergency trains and large-scale maintenance for breaking upgrades.