Two 24K-GPU clusters underpin Meta's Llama 3 training
Meta has detailed the design of its two latest AI training clusters, each built around 24,576 NVIDIA H100 GPUs. The company describes the systems as the hardware foundation for its current and next-generation GenAI models, including Llama 3, and says the clusters are already in use for ongoing training workloads. The design builds on Meta's open contributions to the Open Compute Project and its continued development of PyTorch.
These clusters represent a scale-up from Meta's Research SuperCluster (RSC), the 16,000-GPU A100 system introduced in 2022 that trained the original Llama and Llama 2 models. The new capacity is part of a broader infrastructure roadmap: Meta says it aims to have compute power equivalent to nearly 600,000 H100s by the end of 2024, including 350,000 H100 GPUs within that portfolio.
Cluster architecture and networking choices
The two clusters share the same GPU count but differ in their network fabric. One uses remote direct memory access (RDMA) over converged Ethernet (RoCE), built on the Arista 7800 series with Wedge400 and Minipack2 OCP rack switches. The other uses an NVIDIA Quantum2 InfiniBand fabric. Both interconnect 400 Gbps endpoints.
Meta says the dual-fabric approach lets it evaluate scalability and suitability of each interconnect type for large-scale training. Through co-design of the network, software, and model architecture, the company reports that both RoCE and InfiniBand clusters have handled large GenAI workloads without network bottlenecks — including Llama 3 training, which runs on the RoCE cluster.
Compute and power design
Both clusters use Grand Teton, Meta's in-house GPU hardware platform contributed to OCP. Grand Teton integrates power, control, compute, and fabric interfaces into a single chassis, which the company says improves performance, signal integrity, and thermal behavior. Combined with Meta's Open Rack power architecture, the platform allows rapid deployment and simplified maintenance. Grand Teton follows a line of open GPU designs that dates back to the Big Sur platform from 2015.
Storage architecture
Storage is a frequently overlooked component of AI training infrastructure, and GenAI workloads with multimodal data make it increasingly demanding. Meta's cluster storage uses two approaches.
The first is a home-grown Linux Filesystem in Userspace (FUSE) API backed by a Flash-optimized version of Meta's Tectonic distributed storage solution. This supports thousands of GPUs synchronously saving and loading checkpoints — a challenge for any storage system — while offering the exabyte-scale throughput needed for data loading.
The second is a parallel network file system (NFS) deployment co-developed with Hammerspace. This enables interactive debugging for large multi-GPU jobs, since code changes propagate immediately across all nodes. Both storage tiers run on the YV3 Sierra Point server platform with high-capacity E1.S SSDs.
Performance tuning and scale-up results
Meta's approach to validating large-cluster design is empirical: build, optimize, and test. Initial all-gather collective performance on the large clusters was poor and inconsistent compared to small, optimized clusters. The team addressed this via several changes, including topology-aware job scheduling to reduce traffic to upper network layers, and network routing optimizations combined with NVIDIA Collective Communications Library (NCCL) changes.
Software optimization extended beyond network infrastructure. Meta worked with training framework authors to exploit H100's 8-bit floating point (FP8) data type. New parallelization techniques and storage improvements enabled checkpointing across thousands of ranks in hundreds of milliseconds.
The company also identifies debuggability as a major issue at this scale. It is building distributed debugging tools, including a distributed collective flight recorder, to expose the state of distributed training and help quickly identify problematic GPUs. Additionally, PyTorch work trimmed process group initialization times in some cases from hours to minutes, preparing the framework for tens or hundreds of thousands of GPU training runs.
Open innovation continues
Meta frames these clusters as part of its ongoing commitment to open AI ecosystems — both hardware and software. The hardware designs, including Grand Teton and Open Rack, are contributed to OCP. Meta remains the primary contributor to PyTorch, which powers a significant share of industry AI workloads.
Beyond infrastructure, the company cites its Open Innovation AI Research Community for academic partnerships focused on LLMs, and its participation in the AI Alliance, a coalition of organizations advancing responsible AI development within an open community.
Scaling toward the next generation
Meta’s two AI training clusters represent a step in a broader infrastructure roadmap that extends through the end of 2024. The company’s goal is to reach a portfolio of 350,000 NVIDIA H100 GPUs, with total compute power equivalent to roughly 600,000 H100s.
The design philosophy behind these clusters reflects an awareness that current solutions may not suffice for future workloads. Meta is continuously evaluating and refining its infrastructure—encompassing physical hardware, virtualization, and software—to build systems that remain flexible and reliable as new models and research requirements emerge.
