Meta Engineers Share QUIC and TCP Wins from Production Networks
QUIC and HTTP/3 now account for more than 75 percent of Meta's internet traffic, positioning QUIC as the de facto protocol for internet communication across the company's products. In data center networks, TCP remains the primary transport protocol, supporting thousands of services on top of it. At the recent Networking @Scale 2022 virtual conference, engineers from Meta presented several production deployments aimed at improving performance, congestion management, and platform extensibility across CDN, edge, backbone, WAN, and data center layers.
QUIC-Based Direct Server Return at the CDN Layer
Meta engineers Matt Joras and Yair Gottdenker described a solution that leverages QUIC's properties at the CDN layer to implement direct server return (DSR) from the caching layer directly to the client. In a typical CDN architecture, serving cached content can involve streaming through multiple hops and significant intracluster communication. The QUIC-based DSR approach bypasses most of that internal traffic, resulting in substantial CPU cycle savings and reduced intracluster network bandwidth consumption. The talk covered implementation details, observed performance improvements, and future applications for the technique.
Jump-Starting QUIC Transfers on High-BDP Links
Transfers over high-bandwidth-delay-product (BDP) links face a startup phase where congestion control must probe to discover available bandwidth. For small transfers, this probing can consume the entire transfer time, preventing the connection from ever reaching full utilization. Research Scientist Joseph Beshay presented an application of QUIC in Meta's backbone network where congestion control state is cached and reused to "jump-start" new connections. This approach significantly reduces the startup delay on high-BDP links by allowing new connections to begin with previously learned congestion state rather than probing from scratch.
Addressing Data Center Congestion and Bursts
With Meta's data center network growing alongside its user base, engineers have focused on ensuring high reliability and performance. Abhishek Dhamija and Balasubramanian Madhavan discussed two transport tuning initiatives deployed in data centers:
- Sustained congestion handling using DCTCP, which relies on ECN-based congestion signals
- Burst mitigation through receiver window tuning
The presentation covered motivation, implementation overview, and the coexistence of multiple congestion control mechanisms in the DC environment. BPF-based enablement knobs were used to manage which mechanism applies to which traffic, alongside the wins and lessons learned from both initiatives.
NetEdit: A Platform for Fine-Grained Network Tuning
Large-scale network changes at Meta must be made without impacting production traffic, so every modification needs thorough development, validation, and testing before rollout. Prashanth Kannan and Prankur Gupta presented NetEdit, a BPF-based network feature platform designed for extensibility and modularity. NetEdit is stateless and embeds monitoring and observability at its core, enabling engineers to tune network transport across millions of servers efficiently.
WAN Entitlement: From Hose-Based Approval to Host-Based Admission
Meta's WAN interconnects multiple data center regions and hundreds of points-of-presence (POPs), with resources shared by several high network demand services. Building an overprovisioned network to accommodate all service peak demands simultaneously is impractical due to fiber sourcing, deployment constraints, and cost. Guanqing Yan and Manikandan Somasundaram presented Meta's production traffic classification and WAN entitlement framework, which provides a stable, operations-friendly abstraction for sharing the backbone safely.
The framework has two key components:
- A hose-based entitlement granting system that establishes an agile capacity contract while maintaining network efficiency and long-term SLO guarantees
- A distributed host-based traffic admission system that enforces the contract on production traffic at scale
