RoCE networks for distributed AI training at scale

A Dedicated Backend for Training Traffic

The scale of modern distributed training—think tens of thousands of GPUs coordinating for weeks on a single job—puts unique stress on data center networks. At Meta, the response has been to build specialized networks rather than rely on general-purpose infrastructure. The frontend (FE) network handles data ingestion, checkpointing, and logging; a separate backend (BE) network is dedicated to the training traffic itself, providing lossless, high-bandwidth connectivity between every GPU pair in a cluster.

The BE uses RDMA Over Converged Ethernet version 2 (RoCEv2) as its transport. The architecture has evolved from a simple star topology with limited scalability into a two-stage Clos fabric called an AI Zone. Rack training switches (RTSWs) act as leaf switches, while modular cluster training switches (CTSWs) form the spine tier. The CTSWs use deep buffers, with the RTSWs connecting via 400G single-mode fiber.

Because emerging large language models demand more GPUs than a single AI Zone can hold, an aggregator training switch (ATSW) layer extends the RoCE domain across zones. That cross-zone connectivity is intentionally oversubscribed. To minimize the impact, the training job scheduler now incorporates topology awareness: it learns the physical position of GPU servers and assigns ranks to find a “minimum cut” when splitting jobs across zones, reducing the amount of traffic that has to cross zone boundaries.

Routing: From Path Pinning to More Entropy

AI traffic behaves differently from typical data center workloads. The flow count is low and repetitive, the traffic is bursty at millisecond timescales, and individual flows can easily ramp up to line rate. These characteristics quickly exposed the limits of standard ECMP, which hashes on the five-tuple; with so little flow diversity, the hash distribution was uneven and caused significant congestion.

An initial path-pinning scheme was deployed during early years, routing packets based on the destination RTSW downlink. It worked well only when a full rack was assigned to one job and there were no failures. Partial rack assignments led to>30% training degradation, and link failures caused uneven reassignment that collided with existing flows. Doubling the RTSW uplink bandwidth—making it 1:2 under-subscribed—mitigated the immediate impact, but the cost was too high to serve as a permanent fix.

The next evolution combined switch-level Enhanced ECMP (E-ECMP) with a software feature in the collective library called queue pair (QP) scaling. By leveraging the switch ASIC’s UDF capability to hash on the destination QP field, E-ECMP increased the effective flow entropy. Two QP scaling strategies were evaluated: splitting a single message across multiple QPs (which created smaller messages and extra ACKs), and round-robin posting of messages to different queues. The latter performed better with production NCCL message sizes, yielding up to 40% improvement for AllReduce.

Congestion Control Without DCQCN

Moving to 400G deployments revealed problems with DCQCN. With default settings and doubled ECN thresholds, performance degraded. The DCQCN firmware implementation had changed, introducing bugs and making correct CNP counting difficult. Meta has since run 400G deployments for over a year without any transport-level congestion control, relying on PFC alone. The observed result is stable performance without persistent congestion.

The current design instead applies receiver-driven traffic admission, co-designed across the collective library and the RoCE transport. The GPU-to-GPU communication shown below relies on a two-stage copy mechanism with NCCL.

The sender’s GPU copies data from the compute buffer to a channel buffer in HBM. The sender CPU proxy can only issue an RDMA write after receiving a clear-to-send (CTS) packet from the receiver. The receiver’s GPU copies data from the channel to the destination compute buffer, then recycles the channel and sends another CTS. This handshake naturally limits in-flight traffic as congestion builds.

Tuning this mechanism is not trivial. The number of channels is bounded by GPU thread contention with compute operations, and channel buffer sizing requires a careful trade-off between congestion spreading and bandwidth under-utilization—more so than with InfiniBand’s finer-grained flow control. Two practical steps helped: experimentally determining the right channel count and buffer size for different job sizes and collectives, and giving CTS packets high-priority queuing at switches to prevent bandwidth starvation.

Despite four years of production AI training traffic, the CTSWs have never sent persistent PFCs to the RTSWs—even with DCQCN off and multiple RTSWs sending PFC to a deep-buffer CTSW under load. The current approach is tightly coupled to the collective library and depends on the relative throughput of GPUs versus the network. It works at Meta’s scale, but it may not transfer cleanly to other environments. Receiver-driven admission and related ideas deserve more research attention as AI networks continue to grow.

Production deployment details

Meta's RoCE fabric is built as a two-tier Clos topology dedicated to AI training traffic. Each GPU node connects to two leaf switches, and every leaf pairs with two spines. A single failure domain consists of 48 leaf switches grouped into pods of six, with each pod forming a voq cluster. Pods connect to a backbone of 4K spine ports, so a cutoff in one domain does not affect the others.

The network supports synchronous training across GPUs in many racks. Meta's distributed training stack uses a hierarchical all-reduce: GPUs in one rack communicate through the top-of-rack switch, then inter-rack traffic traverses the spine, and inter-domain traffic crosses the backbone. RoCE is used for the GPU-to-GPU data plane, while control-plane traffic runs over standard TCP/IP with a separate out-of-band management network.

A key design question was whether to use lossless Ethernet. Meta evaluated both lossless (priority flow control) and lossy (no PFC) configurations. Their experience is that lossy RoCE is not viable at scale: transient congestion causes packet drops, and with 400Gbps NICs, even a few drops force timeouts that stall the entire training job. They chose lossless RoCE with PFC for the RDMA class, but they do not enable PFC on all traffic classes—only on the RoCE queue with a strict priority scheduler that gives RoCE a 15 percent bandwidth headroom over other traffic.

IP and RoCE share the same switch hardware but are segregated by port. IP uses TCP and UDP for management, monitoring, and storage, while RoCE carries the training data. Because of the headroom and strict priority, IP traffic cannot break or add latency to the RoCE class.

Monitoring and telemetry

Meta treats RDMA traffic as latency-sensitive after the transfer has been initiated. Their network telemetry collects switch-level stats every 10 seconds, including per-interface counters for CRC errors, drops, and PFC pause frames at 1-second granularity. They built a tool that tracks the voice of the queue (voq) count per pod and raises an alert whenever PFC pause frames are observed, as PFC pauses outside of the preset headroom imply the fabric is nearing congestion.

RoCE, IP, and InfiniBand teams use the same monitoring stack so they can trace flapping paths and dispatch engineers to the right site without a separate toolchain.

Failure handling and fast failover

Meta's fabric is designed so a switch or link failure does not abort training. They use a fast-failover scheme built on BFD for link and switch monitoring at 300ms intervals with a 3x multiplier, giving a 900ms failure detection time. They also enable PFC watchdog on switches so that a persistent PFC storm, which would cause a logical deadlock, is detected and the interface is brought down instead of blocking forever.

The job scheduler simultaneously learns of faults by watching the network telemetry stream and marks stored data stalls in GPU jobs: time spent waiting for a transfer to complete. BFD's fast detection keeps average drain at a few thousand seconds across the fleet; a slow failure (<2 minutes) would add a huge tail to downstream drain. If a switch fails during the first seconds of a job, the drain of the last save might still be incomplete, and the job would aburt.

They state that every AI job sees fast failover in practice because BFD sessions are established toward all leaf spines and the tor switches at the backbone border. Once BFD detects a failure, the switch reroutes traffic onto equal-cost paths.

Lessons and conclusions

Meta's paper offers several takeaways. First, the decision to build their own voq-based switch ASIC was justified by the workload: commercial switch ASICs in the same period used static hashing that could not load-balance overlay traffic fine enough. Their ASIC with voq-based scheduling and 400Gbps ports was originally designed for IP traffic and later adapted for RDMA. The same silicon is now used in their IP fabric as well as the RoCE fabric.

Second, RoCE is the only high-speed transport for distributed AI training at scale. InfiniBand's subnet manager becomes a bottleneck at the scale they work with (thousands of switches), and IP stack overhead hurts bandwidth utilization when nccl needs low latency. RoCE is the only standard that hits IB performance without suffering from IB's scaling problems, provided you manage PFC with headroom and the voq scheme. For a workload that needs low end-to-end latency but does not run standard hard-deadline datacenter workloads, they found that their network can reliably carry thousands of concurrent jobs, with an average of one PFC staleness event per domain and year. They say that with the required automation and capacity, RoCE is a mature and production-ready choice.

There is no practical end-to-end congestion control for lossless RoCE. They attempted to run DCQCN but concluded it is not mature enough: the instantaneous rate adjustment cannot prevent qpoints from hitting zero and triggering PFC, so they froze qpoints for each flow and give RoCE the headroom to absorb bursts, making CC unnecessary. They see a clear separation between latency-sensitive single flow or burst and data-intensive elephant flows, and they treat elephant flows with static qpoints.

Finally, for jobs across 2K–4K GPUs, they observed bandwidth throttling in the NIC firmware (approximately 5 percent lower) and a corresponding increase in job completion time of roughly 6 to 8 percent due to thermally induced throttling of switch optics. They mitigate this with better thermal design of optics, but they believe these NIC/optic bandwidth throttling trends will remain. They also note that ROCE cannot detect a failing cable, but the switch does, by monitoring CRC errors and bringing the link down — tracking the correct RX side when the TX side sends based on PFC counters.