BBRv2 on the Edge: A Field Test
BBRv1 has been a workhorse since its release several years ago, finding its way into Linux, FreeBSD, and Chrome's QUIC stack. On Dropbox's edge network it delivered impressive throughput gains, but it also brought a set of known tradeoffs. These included fairness problems against loss-based congestion controls like CUBIC, RTT-unfairness between BBR flows, and a general disregard for packet loss. The BBR developers and the IETF ICCRG group cataloged the key shortcomings: low throughput for Reno/CUBIC flows sharing a bottleneck with BBR, loss-agnostic behavior, lack of ECN support, poor performance on aggregated paths like Wi-Fi, and throughput variation from low cwnd in PROBE_RTT.
BBRv2 was designed to address these issues, adding network modeling with aggregation and in-flight parameters, and experimental ECN support. To see how it behaves outside the lab, we deployed it on a set of machines in our Tokyo PoP, handling real-world traffic across millions of connections. This was not a single-connection test; the results below come from connection-level ss samples, /proc stats, and nginx logs. We focused strictly on high-throughput workloads, filtering for connections transferring at least 1 Mb.
Test Configuration
We ran four combinations of kernels and congestion control algorithms:
- 5.3 kernel, cubic
- 5.3 kernel, bbr1
- 5.3 kernel, bbr2
- 4.15 kernel, bbr1
All servers used a mix of mq and sch_fq qdiscs with defaults, on a kernel based on Ubuntu-hwe-edge-5.3.0-18.19_18.04.2, patched with the v2alpha-2019-11-17 BBR tag. We also applied a patch adding a TCP_INFO counter for out-of-order packets and a small fix to silence a spurious BBRv2 warning.
Keeping the kernel current matters. Comparing 4.15 to 5.3 shows roughly 15% higher throughput on the newer kernel, likely due to patches like "tcp_bbr: adapt cwnd based on ack aggregation estimation" and the switch to the Early Departure Time model. The 5.3 kernel also carries mitigations for recently discovered CPU vulnerabilities, which we kept enabled despite the CPU cost.
Userspace also needs to keep pace. A recent iproute2 gives ss full access to the kernel's struct tcp_info and the BBRv2-specific struct tcp_bbr_info, exposing useful fields like sender buffer and receive window stats.
Field Observations
Packet Loss and Reordering
The first clear signal after enabling BBRv2 was a significant drop in retransmissions. It didn't reach CUBIC's levels, but the improvement over BBRv1 was immediate. Looking at ss stats on a logarithmic scale, BBRv2's packet loss sits well below BBRv1, though still above CUBIC.
There were some outliers: BBRv2 connections with more than 60% packet loss, which didn't appear for BBRv1 or CUBIC. These didn't correlate with OS, connection type, or RTT. Outside of those anomalies, retransmissions were lower across all RTTs with BBRv2. Hosts also saw less packet reordering, possibly a side effect of having fewer segments in flight.
Throughput and Goodput
For the Traffic team, the key SLI is file download speed as seen from the server. Across the lower percentiles, BBRv2 tracks closer to CUBIC; at the higher percentiles, it moves toward BBRv1's performance. So yes, BBRv2 is slower than BBRv1 on raw bandwidth, but it's still ahead of CUBIC. The tradeoff is better goodput, since fewer packets are wasted on retransmissions.
In exchange for that throughput reduction, we get significantly fewer unacknowledged packets in flight.
Plotting RTT against in-flight data shows a notable shift: BBRv1's in-flight amount tends to scale with RTT, while BBRv2 keeps it fairly flat. As Neal Cardwell explained regarding BBRv1 RTT unfairness, each flow's cwnd is roughly 2bwmin_rtt, which tends to hold one bwmin_rtt in the bottleneck queue. Flows with a higher min_rtt therefore hold more queue and claim more bandwidth. BBRv2 appears to decouple that.
Receiver Window and RTT
BBRv2 connections spent much less time being receive-window-limited than either BBRv1 or CUBIC, which suggests better use of the available window. As a bonus, BBRv2 also achieved lower RTT than BBRv1, though still slightly above CUBIC.
Plotting min RTT against bandwidth shows vertical bands for network distance and horizontal bands for typical internet speeds, with an exponentially decaying relationship for both BBR versions. That suggests bandwidth is sometimes artificially capped, likely by the user's receive window. The 130+ ms RTT band indicates cross-Pacific traffic, which may point to GSLB routing issues covered in a separate post.
CPU Usage
BBRv2's more sophisticated model might sound like it would cost more CPU, but it adds a fast path that skips model updates when the connection is application-limited. That's a big deal for typical workloads, where ACKs arrive in volume. In our testing, though, we saw no measurable CPU difference between BBRv1 and BBRv2. That's likely because this BBRv2 build still has a fair amount of debug code compiled in.
It's worth paying attention to CPU if you test BBR with ECN enabled, since it can disable GRO/GSO in high packet loss scenarios.
What the Numbers Show
Across Dropbox's edge network tests, BBRv2 delivered a clear set of results when compared against its predecessor and CUBIC:
- Bandwidth matches CUBIC for users at the lower percentiles of Internet speeds.
- Bandwidth matches BBRv1 for users at the higher percentiles.
- Packet loss is 4x lower than BBRv1*, but still 2x higher than CUBIC.
- Data in-flight is 3x lower than BBRv1 and slightly lower than CUBIC.
- RTTs are lower than BBRv1, though still above CUBIC's levels.
- RTT-fairness is improved compared to BBRv1.
* Excluding the 0.0001% of outliers experiencing over 60% packet loss.
Overall, BBRv2 represents a major step forward from BBRv1 and looks far closer to a viable drop-in replacement for Reno/CUBIC in scenarios where a bit more bandwidth is desirable. With experimental ECN support added, it could even serve as a drop-in replacement for Data Center TCP (DCTCP).
Inside BBRv2's Design and Development
BBRv2's development is public, hosted on github.com/google/bbr, with discussion on the bbr-dev mailing list. The algorithm's design principles, as presented to the IETF, highlight what's new in v2:
- Leave headroom: leave space for entering flows to grab
- React quickly: using loss/ECN, adapt to delivery process now to maintain flow balance
- Don't overreact: don't do a multiplicative decrease on every round trip with loss/ECN
- Probe deferentially: probe on a time scale to allow coexistence with Reno/CUBIC
- Probe robustly: try to probe beyond estimated max bw, max volume before we cut est.
- Avoid overshooting: start probing at an inflight measured to be tolerable
- Grow scalably: start probing at 1 extra packet; grow exponentially to use free capacity
Key milestones in the v2 rollout have tracked its evolution from algorithm description to open-source release and subsequent performance fixes at IETF 104 (Prague, Mar 2019), IETF 105 (Montreal, July 2019), and IETF 106 (Singapore, Nov 2019).
The State of ECN
Explicit Congestion Notification (ECN) allows a network bottleneck to signal senders to slow down before buffers are exhausted and packet drops begin. However, current Internet deployments largely treat ECN in a "passive" mode. At Dropbox's Tokyo point of presence, only 3.68% of connections negotiate ECN, and of those, 88% carry the ecnseen flag. Apple's data shows more than 74% of popular websites support ECN passively.
Classic ECN (RFC3168) has a significant drawback: it is overly prescriptive about the explicit congestion signal. This restriction exists because any behavioral difference between explicit congestion (CE mark) and implicit congestion (drop) would inevitably produce unfairness. RFC7560 further highlights the low granularity of classic ECN, which can only provide one congestion signal per RTT. This matters because both DCTCP and BBRv2 benefit greatly from more precise ECN feedback. In fact, DCTCP's custom interpretation of CE causes total unfairness toward classic congestion-control algorithms — a problem RFC8311 attempts to address by relaxing the restrictions on ECN experimentation.
Beyond implementation issues, there is an ongoing standards conflict over a single codepoint (ECT(1)) in the IP header. The L4S (Low Latency, Low Loss, Scalable Throughput) proposal and the SCE (Some Congestion Experienced) draft offer incompatible interpretations. As Jonathan Corbet summarizes, SCE argues its use is fully compatible with existing deployments, while L4S would reserve the value for protocols incompatible with current congestion control. L4S proponents maintain that a dual-queue architecture is essential for achieving their latency targets.
Pacing and Fair Queuing
Dropbox's edge boxes run with the Fair Queue (fq) scheduler, but not primarily for the fair-queueing itself — the real goal is the pacing that tc-fq introduces. Earlier fq implementations added jitter to TCP's RTT estimation, a problem for data-center RPC p99s. That issue was resolved in the kernel change "tcp: switch to Early Departure Time model." Using bpftrace, engineers can observe the difference between when packets are enqueued to the qdisc versus when they are dequeued, illustrating pacing in action.
This aligns with the broader argument that sending "as fast as possible" is no longer optimal, as articulated in Van Jacobson's talk "Evolving from AFAP – Teaching NICs about time." The talk makes a strong case for network-layer pacing and delay-based congestion control. Where possible, deploying active queue management (AQM) like fq_codel or cake at bottleneck points can further improve the situation.
