Training LLMs at Meta’s scale

Meta’s AI training workloads have changed shape in a fundamental way. Recommendation models and ranking systems demand a large number of smaller jobs spread across comparatively few GPUs. Generative AI flips that model: fewer jobs, but each one consumes an enormous number of GPUs and must run as a single, synchronized job on a shared high-speed fabric.

That shift introduces a set of operational problems that Meta had to solve across software, hardware, and network design.

What breaks at 10,000+ GPUs

The core issue is statistical: the more GPUs in a job, the higher the chance that something fails mid-training. Because all GPUs in a job must communicate constantly, a single slow or dead device can stall the entire run. Meta groups the necessary work into four areas:

  • Hardware reliability: Minimize the chance of failure with testing, quality control, and automated detection and remediation.
  • Fast recovery on failure: Reduce re-scheduling overhead and speed up training re-initialization so downtime stays short.
  • Efficient preservation of training state: Regularly checkpoint training state and store/retrieve it efficiently so jobs can resume from where they stopped.
  • Optimal connectivity between GPUs: Ensure synchronized data exchange doesn't get bottlenecked by a subset of slow links.

Reworking the infrastructure stack

Training software and scheduling

Meta lets researchers work with PyTorch and other open source tools to shorten the path from research to production. Scheduling algorithms allocate resources dynamically to adapt to changing workloads and keep GPU utilization high.

Hardware and data center changes

With hardware development cycles too long for the GenAI timeline, Meta adapted existing systems. The Grand Teton platform, originally built around NVIDIA H100 GPUs, was modified to increase GPU TDP to 700W and move to HBM3 memory. Cooling infrastructure couldn't be changed in time, so the platform stayed air-cooled, forcing changes to mechanical and thermal design and triggering a full validation cycle before large-scale deployment.

In the data center, power and cooling constraints can't be changed quickly. Meta optimized by relocating supporting services (like readers) out of the data hall and packing as many GPU racks as possible into each hall to maximize compute density and network capability per cluster.

Reliability in practice

Failure count scales with cluster size, so Meta keeps spare capacity to restart jobs quickly and monitors for failures that can be prevented before they interrupt training. The most common failure modes observed so far:

  • GPUs falling off: GPUs not detected by the host over PCIe. More frequent early in a server's life, then settles with age.
  • DRAM & SRAM UCE: Uncorrectable memory errors. Meta monitors repeat offenders, tracks them against thresholds, and initiates RMAs when error rates exceed vendor limits.
  • HW network cable: Falls under the broader "unreachable servers" category, also more common early in a server's life.

Choosing a network fabric

Large-scale training requires moving huge volumes of data between GPUs quickly. Two industry-standard options fit: RoCE and InfiniBand. Meta had four years of experience running RoCE clusters, but only at up to 4K GPUs, while its InfiniBand research clusters reached 16K GPUs but weren't integrated into the production environment or built for the latest generation of GPUs.

Rather than picking one, Meta built both: two 24K GPU clusters, one RoCE and one InfiniBand, with the RoCE side optimized for quick build time and the InfiniBand side for full-bisection bandwidth. Both were used to train Llama 3, with the RoCE cluster training the largest model. After tuning, both fabrics delivered equivalent performance for large GenAI workloads despite their different underlying technologies.

Three optimizations made network communication performant on both fabrics:

  1. Communication patterns from model, data, and pipeline parallelism were mapped to different network topology layers to exploit available capabilities.
  2. Collective communication was made topology-aware. Default ring-based implementations were replaced with custom algorithms like recursive doubling or halving, which are less latency-sensitive.
  3. Additional load-balancing and routing work was needed to distribute traffic evenly across all paths, since GenAI jobs produce large "fat flows" that are hard to spread across the network.

Storage

Training data volumes require high-capacity, high-speed storage, plus new storage solutions tailored to the specific access patterns of GenAI workloads.

What’s next

Meta expects to run jobs across hundreds of thousands of GPUs in the coming years, with larger data volumes and longer network distances to manage. New GPU architectures and other hardware will drive further changes to the infrastructure stack. The specific engineering problems that creates aren't fully predictable, but they will keep pushing changes across all layers of the stack.