Meta Details the Network Stack Behind Its AI Workloads

Meta’s push into large-scale AI — spanning custom silicon like MTIA, public models such as Llama 2 and Code Llama, and a growing portfolio of GenAI features — depends on infrastructure that can move data at unprecedented scale. At the 2023 Networking at Scale conference, engineers walked through how Meta has built and operated the networks that carry its ranking, recommendation, and generative AI training workloads.

The sessions covered the full lifecycle of Meta's AI network infrastructure: physical and logical fabric design, custom routing and traffic engineering, performance tuning and observability, and simulation tools for planning future systems. A common thread was the shift from CPU-based to GPU-based training and the need to keep thousands of accelerators working in lockstep.

RoCE Fabrics and the Road to Scale-Out Training

Meta's current training infrastructure uses a RoCE-based network fabric with a CLOS topology. Leaf switches connect to GPU hosts while spine switches supply the scale-out connectivity between GPUs in a cluster. Engineers Hany Morsy and Susana Contrera detailed the evolution of these builds as workloads demanded more distributed, network-interconnected systems.

In a companion session, network engineer Adi Gangidi gave a deeper look at Meta's RDMA deployment, which relies on RoCEv2 transport for production AI training. The talk focused on how the design is positioned to maximize raw performance and consistency — two properties that are critical when training jobs depend on tightly synchronized parallel operations. Gangidi also covered challenges in routing, transport, and hardware that Meta addressed as the infrastructure grew, along with areas where further work is expected over the next few years.

Traffic Engineering for Consistent Job Performance

Meta has run RoCE-based distributed training clusters internally since 2020, but early on these systems struggled with inconsistent job performance. To solve this, software engineers Shuqiang Zhang and Jingyi Yang described Meta's centralized traffic engineering approach, which dynamically places traffic across all available paths in a load-balanced manner. Their session covered the design, implementation, evaluation, and operational history of the system.

New Collective Communication Benchmarks and Observability

Reliable collective communication across Meta's AI-Zone RDMA network is foundational for scaling training and inference. Research scientist Shengbao Zheng discussed how Meta built top-down observability that ties workload-level performance to the network, making it possible to attribute training failures or slowdowns to backend issues when appropriate.

Two tools support this effort:

  • ROCET — associates jobs with RDMA network metrics and provides analysis on top of that data.
  • PARAM — a benchmark for analyzing and tuning collective communication operations using workload traces.

Meta has shared these systems with the broader community through the Chakra ecosystem, which it says enables co-design of more efficient distributed ML systems.

Simulation for Future AI Systems

Research scientist Zhaodong Wang and networking modeling and optimization engineer Satyajeet Singh Ahuja presented Arcadia, a unified simulation system that models the compute, memory, and network performance of AI training clusters. The tool provides a multidisciplinary performance analysis framework that supports design and optimization across the application, network, and hardware layers.

Arcadia serves two primary functions at Meta:

  • Simulating how future AI models and workloads will perform on specific infrastructure, informing decisions about model design and hardware evolution.
  • Modeling the impact of scheduled operational tasks on production AI models, helping engineers make job-aware decisions during daily operations.

The broader theme across all sessions was that GenAI model requirements are driving changes not just in Meta's hardware and software, but in the fundamental network architecture that connects them. The talks offered a look at how Meta is addressing the unique challenges that come with pushing distributed training to larger and larger scale.