Meta’s networking engineers tackle scale at every layer
Operating a network at Meta’s scale produces a predictable set of tensions: proprietary switch hardware with inconsistent feature sets, data-center builds consuming megawatts per physical facility, and traffic patterns reshaped by AI, HPC, and ML workloads that must coexist with legacy web services. Content growth and hybrid work have similarly strained backbone capacity, while traditional metrics like packet loss and jitter fail to connect network behavior to application-level performance.
At the November 2022 Networking@Scale conference, engineers walked through how they approached these problems across data center switching, submarine and terrestrial transport, traffic visibility, service-level objectives, and load-balancer behavior.
FBOSS: in-house switch software at brownfield scale
FBOSS, Meta’s in-house software for managing data center switches, is one of the largest services in the company’s network. In their session, Shrikrishna Khare and Srikrishna Gopu described the design principles that let them build a stable control plane over heterogeneous forwarding hardware.
Key to the effort is the Switch Abstraction Interface (SAI), a vendor-independent API for programming forwarding ASICs. Meta deployed the new FBOSS implementation into a brownfield environment at massive scale, and used it to bring a new switch vendor onto the existing infrastructure.
The global network: submarine systems and optical hierarchy
Backbone Engineering plans, designs, builds, and operates the wide-area network connecting Meta’s data centers and points-of-presence. Stephen Grubb and Joseph Kakande covered the scale of that build, with emphasis on submarine fiber.
The Bifrost and Echo systems mark the first direct subsea connections between the US and Singapore, and will support SGA, Meta’s first APAC data center. The 2Africa project, meanwhile, is the largest submarine cable network planned anywhere, with potential reach to roughly 3 billion people. The session also covered how subsea segments tie into the terrestrial backbone, how Meta structures the optical transport hierarchy over those fiber paths, and the in-house software used to provision and monitor distributed hardware fleets and to diagnose and fix network faults.
Millisampler: fine-grained traffic visibility
Data center networks with low loss depend on understanding traffic patterns, especially burstiness at fine time granularity. Fleet-wide monitoring at millisecond granularity had been impractical, so Yimeng Zhao presented Millisampler, a BPF-based traffic measurement tool deployed on every server in the fleet for continuous monitoring.
Millisampler captures microbursts at millisecond or even microsecond resolution. Because collection is simultaneous across servers, the data shows how synchronized bursts interact in rack buffers. Zhao covered the tool’s design, implementation, production experience, and notable observations drawn from the collected data.
Network SLOs: separating network faults from app behavior
Determining whether a slow image load or stalling video stems from network conditions is a persistent operational problem. Brandon Schlinker and Sharad Jaiswal presented Network SLOs—each product’s minimum network requirements for acceptable quality of experience—as a way to answer that question routinely.
Their session described deriving these SLOs with statistical tools, operationalizing them, and evaluating compliance. Case studies showed the SLOs being used to triage QoE regressions, expose gaps in edge network capacity, and reveal ways product code underutilizes the network.
Shiv: consistent L4 routing during backend churn
Large-scale additions or removals of backends trigger remappings in routing tables, which can break end-to-end connections and degrade experiences such as video playback. Aman Sharma and Andrii Vasylevskyi presented Shiv, a tool developed to improve Layer 4 routing consistency.
Shiv selects a backend using a consistent hash of the packet’s 5-tuple: source IP, destination IP, source port, destination port, and protocol. The objective is to keep all packets for a given connection—which share the same 5-tuple—on the same backend for the connection’s lifetime, preventing breakage during remapping events.
