RDMA With a Different Center of Gravity

Meta's training and inference clusters now span hundreds of thousands of GPUs across multiple data centers, and the network sits directly in the critical path of both. Collective operations like all-reduce and all-to-all synchronize thousands of accelerators, where the slowest transfer paces the entire job. In inference, latency between distributed model shards translates directly into user-facing response times.

MetaRoCE is Meta's clean-sheet RDMA transport protocol built for AI workloads on commodity Ethernet at million-GPU scale. The specification, a reference software implementation, and a compliance test suite are being released through the Open Compute Project (OCP).

The protocol's central design decision is to push intelligence from the fabric to the endpoint. Standard RoCE expects in-order delivery and relies on PFC and lossless fabric behavior. MetaRoCE instead decomposes the network into fine-grained logical paths, each carrying its own real-time telemetry on RTT, ECN state, and utilization.

Out-of-Order as the Default

MetaRoCE sprays packets across many paths, making out-of-order arrival the normal case rather than an exception. Every packet is self-contained: writes carry their destination address, and sends carry the match to a posted receive buffer. Data lands directly in its final memory location without a reorder buffer or head-of-line blocking. Two-sided messaging is available where collective libraries prefer it instead of forcing everything into writes.

Paths Are First-Class Objects

Each connection in MetaRoCE maintains multiple paths and sprays packets across them. Every path uses a distinct UDP source port as ECMP entropy, which the NIC can change to move traffic off a failing route. On multiplane fabrics, the NIC owns plane selection entirely. Each path keeps its own window and round-trip estimate, letting the transport distinguish congestion from link failure and rebalance explicitly.

The fabric is treated as inherently lossy. MetaRoCE requires no PFC and sends no pause frames. Each path carries its own ordered sequence, so a gap in the 256-bit selective acknowledgment bitvector signals loss, not reordering. Retransmission targets exactly the missing packet on the path that dropped it.

Congestion Control From Both Ends

Congestion control combines sender-driven, ECN-based AIMD with receiver-driven rate hints. Windows are maintained per path and per connection, so a congestion mark trims only the affected path and steers subsequent packets toward clear ones. Every acknowledgment carries the receiver's allocated share of inbound bandwidth for that sender, letting senders converge on the correct rate directly rather than probing for it. Incast resolves within one or two round trips.

What the Fabric Must Provide

MetaRoCE asks only for ECN marking and ECMP, both already present in standard switches. It requires no packet trimming, in-network telemetry, credit-based flow control, or switch-side spraying—and it still functions if those features are present. The same transport runs over fat-tree, multiplane, deep-buffer, and shallow-buffer fabrics, including third-party clouds where fabric configuration is outside operator control.

The queue pair model is also reworked. Traditional RDMA opens dozens of QPs per node pair to scale ordered streams and bandwidth, each with independent congestion state on the NIC. MetaRoCE separates the concerns: one connection carries many ordered streams above and many paths below under a single congestion controller. Connection state no longer grows with workload parallelism. Existing RDMA Verbs APIs and software stacks continue to work without modification; extension APIs expose enhanced features such as multiplane support.

Validation Results

Meta worked with AMD to implement MetaRoCE on Pensando programmable NICs for hardware validation. On a 64-node AMD GPU cluster running RCCL collectives, MetaRoCE was compared directly against RoCEv2 across all-reduce and all-to-all operations.

MetaRoCE showed consistently higher throughput and lower flow completion times than RoCEv2. Under packet loss that would degrade RoCEv2, MetaRoCE held approximately 86% throughput at 1% loss and continued delivering useful bandwidth even at 10% loss. Multiplane validation across 4-plane and 8-plane topologies with up to 4,000 concurrent connections showed throughput scaling linearly with plane count. Simulated plane failures triggered graceful, autonomous recovery with no application involvement or operator intervention.

Open Contributions

MetaRoCE extends the OCP's Ethernet Scalable Unified Network (ESUN) initiative from the fabric into the transport layer. The contributions to OCP include the full protocol specification for any vendor to implement, a production compliance suite for proving implementations match the spec, and libsoftmetaroce, a reference software transport stack running on commodity Linux over standard UDP sockets. The software library serves as the behavioral model for silicon development and the basis for the compliance framework.

The design is intended to run on both programmable and fixed-function NIC architectures, with the Pensando implementation already proven and other vendor implementations underway.

Remaining Work

Scale-out networking within the data center is the primary focus so far, but other distance and latency regimes remain under active development:

  • Scale-up: Within a rack, accelerators exchange small messages where nanoseconds matter. MetaRoCE has removed the reorder buffer and PFC latency; work continues on optimizing the fast signaling path for short memory operations issued directly between processing elements.
  • Scale-across: Jobs spanning buildings thousands of kilometers apart face millisecond round trips where small path differences compound. The first-class path model already adapts by preferring uncongested routes; the open problem is fairly sharing contended long-haul links.
  • Storage and KV-cache: Distributed storage generates incast when one read fans out to many servers. Receiver-driven rate hints let the receiving side—a storage server taking writes or a client taking reads—moderate inbound rate regardless of fan-out. Keeping that rate accurate across varying network speeds and request sizes is the active challenge.

The MetaRoCE specification, a DPDK-optimized software reference implementation, and the production compliance framework are scheduled for release at the 2026 OCP Global Summit in October.