Meta brings its AI network fabric to OCP
At OCP Summit 2024, Meta is detailing the next-generation network fabric designed for its AI training clusters. The company is contributing two disaggregated network fabrics and a new NIC design to the Open Compute Project, continuing a collaboration that dates back to OCP's founding in 2011. The goal is straightforward: break traditional data center technologies into interchangeable components that can be mixed across vendors, making large-scale AI infrastructure more flexible, scalable, and efficient.
Disaggregated Scheduled Fabric
The centerpiece of the announcement is the Disaggregated Scheduled Fabric (DSF), built for Meta's next-generation AI clusters. DSF applies disaggregation principles to VoQ-based switched systems controlled by the OCP-SAI standard and FBOSS, Meta's open-source network operating system. Unlike fabrics that react to congestion after it occurs, DSF uses VoQ-based traffic scheduling for proactive congestion avoidance, letting Meta build large, non-blocking fabrics that can support high-bandwidth AI workloads.
The fabric exposes a standard Ethernet-based RoCE interface to endpoints and accelerators, including Meta's MTIA and xPUs and NICs from multiple vendors.
Two hardware platforms anchor the DSF design:
Arista 7700R4 series — Dedicated leaf and spine systems combine to form a large distributed switch. The 7700R4C-38PE leaf switch, based on Broadcom Jericho3-AI, offers 18 x 800GE (36 x 400GE) OSFP800 host ports, 20 x 800Gbps fabric ports, and 14.4Tbps of wirespeed performance with 16GB of buffers. The 7720R4-128PE spine switch, built on Broadcom Ramon3, provides 128 x 800Gbps fabric ports and 102.4Tbps of wirespeed performance with an accelerated compute-optimized pipeline.
51.2T switches for 400G/800G fabrics
Meta will deploy two next-generation fabric switches for its 400G/800G data center networks: Minipack3 and the Cisco 8501. Both are backward compatible with previous 200G and 400G switches. Minipack3 is the latest iteration of Meta's own open fabric switch design and uses Broadcom's Tomahawk5 ASIC; the Cisco 8501 runs on Cisco's Silicon One G200. Each switch delivers 51.2Tbps across 64x OSFP ports, avoids retimers to maximize power efficiency, and significantly reduces power per bit versus earlier models. Both run FBOSS.
Moving to 2x400G optics
Meta's fabric has evolved from 200G/400G to 400G/800G links, and 2x400G FR4 optics are already deployed in its data centers.
Extending FBOSS and SAI for DSF
Meta continues to rely on OCP-SAI to bring the new fabric platforms, switch hardware, and optical transceivers into FBOSS. Working with vendors and the OCP community, Meta has extended SAI to support DSF and enhanced routing schemes. Because these are open systems, engineers outside Meta can contribute software that works with the hardware and share it across the industry.
FBNIC: a multi-host NIC with a Meta-designed ASIC
FBNIC is a foundational NIC built around the first Meta-designed network ASIC for its server fleet and MTIA systems. The NIC supports up to four hosts with full datapath isolation per host. Its driver has been upstreamed to the Linux kernel and is available from v6.11. Marvell designed the NIC module, which has been contributed to OCP.
Key features of FBNIC include:
- Network interfaces for up to 4x100/4x50/4x25 GE with SerDes support up to 56G PAM4 per lane
- Up to 4 independent PCIe Gen5 slices
- Hardware offloads including LSO and checksum
- Line-rate timestamping per host, from PHY up, for PTP
- Header-data split for zero-copy support
- Compliance with the OCP NIC 3.0 design specification, version 1.2.0
A vendor-neutral path forward
Scaling AI infrastructure is about more than raw size. Systems also have to remain flexible, power-efficient, and built from components that anyone can adopt or improve. Meta's contributions at OCP — the DSF fabric, the 51.2T switches, and the FBNIC design — are aimed at making advanced AI networking hardware something the wider industry can build on, rather than a proprietary advantage locked inside a single vendor's product line.
