QUIC Congestion Control Gets CUBIC and HyStart++
Cloudflare's quiche, its IETF QUIC implementation, has been running CUBIC congestion control in production for some time. The library has now added HyStart++ to its congestion control module as well. Both mechanisms, well-established in the TCP world, are now available in quiche's pluggable congestion control architecture.
Congestion control determines how much data a connection can send into the network. It must detect the current link capacity in real time and tune itself without overrunning the link or competing unfairly with other connections. QUIC's congestion control draws heavily on TCP experience, using the familiar concepts of CWND (congestion window) and SSTHRESH (slow start threshold).
The QUIC specification recommends Reno as a relatively simple starting point, but the protocol is designed for pluggable congestion control. Since QUIC is implemented from scratch, it can adopt more advanced mechanisms without the legacy constraints of TCP.
Loss-based algorithms like Reno and CUBIC respond to packet loss events, while delay-based approaches like Vegas and BBR try to balance bandwidth and RTT increases. TCP congestion control algorithms can be ported to QUIC with minimal changes by implementing a few hooks; quiche provides a modular API for adding new congestion control modules.
From Reno's Sawtooth to CUBIC's Curve
Reno, the standard congestion control for both TCP and QUIC, is simple and requires no additional memory for state. It starts in slow start mode, roughly doubling CWND every RTT until congestion is detected or CWND exceeds SSTHRESH. After a loss event, it enters recovery mode, then congestion avoidance, where CWND grows by roughly one packet per RTT. This produces the characteristic "sawtooth" pattern in CWND over time.
CUBIC, released in 2008 and now the default in the Linux kernel, differs primarily during congestion avoidance. Instead of linear CWND growth, it uses a cubic function:
When congestion is detected at a CWND value called Wmax, CUBIC reduces the window by 30% (Reno cuts 50%) and then grows according to the cubic function. It approaches Wmax carefully, then enters "Max Probing" to find a new, higher CWND. A "TCP-friendly" mode ensures CWND growth never falls below Reno's. Notably, the original CUBIC specification only defines congestion avoidance behavior; slow start remains identical to Reno.
quiche's CUBIC implementation follows RFC8312, with a fix from the Linux kernel by Google.
HyStart++ Tames Slow Start
CUBIC's authors separately addressed slow start, since CUBIC only changes congestion avoidance. The original HyStart proposed two mechanisms for exiting slow start early: detecting RTT delay increases over a threshold, and ACK train inter-arrival times exceeding a threshold.
In practice, ACK compression made the ACK train approach unreliable, and RTT delay sampling suffers on unstable networks. HyStart++, an IETF draft from Microsoft engineers (now in Windows 10's TCP stack), simplifies the approach:
- Uses only RTT delay sampling, dropping ACK train detection
- Adds a Limited Slow Start (LSS) phase between slow start and congestion avoidance
- LSS grows CWND faster than congestion avoidance but slower than slow start
- Provides a simpler implementation overall
In quiche, HyStart++ is enabled by default for both Reno and CUBIC, and can be configured via the API.
Lab Results: Performance Under Loss
Tests ran in Cloudflare's transport protocol test lab with 5Mbps bandwidth, 60ms RTT, and packet loss rates from 0% to 8%. An 8MB file was downloaded 20 times via NGINX with an HTTP/3 patch, taking the median download time. The comparison included TCP CUBIC (Linux kernel 4.14) and four QUIC configurations: Reno, Reno with HyStart++, CUBIC, and CUBIC with HyStart++.
At 0% packet loss, TCP and QUIC perform nearly identically, with QUIC slightly slower. As loss increases, QUIC CUBIC outperforms TCP CUBIC, confirming that QUIC's loss recovery works well in real-world network conditions.
HyStart++ doesn't change overall download performance, which is expected — its purpose is preventing overshooting the network, not speeding up transfers.
HyStart++ Reduces Packet Loss
The benefits of HyStart++ appear in packet loss reduction. At 0% packet loss (where only network congestion creates loss), average lost packet counts across 20 runs show significant improvement:
CUBIC generally creates more packet loss than Reno because its CWND can grow faster during congestion avoidance, and it reduces the window less (30% versus 50%) at congestion events.
Visualizing Behavior with qlog and qvis
quiche's recent qlog support works with qvis, a visualization tool based on the qlog specification. These tools provide detailed insight into QUIC connection behavior — useful for protocol development and understanding how congestion control algorithms interact.
For a 16MB transfer at 0% packet loss without HyStart++, the congestion chart shows a high CWND peak early from slow start, followed by the characteristic CUBIC concave window growth pattern:
Zooming into the first 0.7 seconds reveals linear CWND increase during slow start, continuing until a packet loss around 500ms triggers recovery and congestion avoidance:
With HyStart++ enabled, the slow start peak is smaller, indicating less overshooting:
Zooming in again shows slow start exiting to Limited Slow Start around 390ms, then moving to congestion avoidance at the congestion event around 500ms. The slope is less steep until congestion is detected, resulting in fewer lost packets and faster convergence to a stable CWND:
What's Next
QUIC's draft specification incorporates extensive TCP experience and recommends Reno to get implementations started, but the protocol's pluggable design supports adopting state-of-the-art mechanisms. CUBIC and HyStart++ deliver better performance than Reno in terms of download speed and packet loss, and quiche now includes both.
Future work includes packet pacing, advanced recovery, and BBR congestion control. The quiche config API allows switching among multiple congestion control algorithms at the connection level, and qlog endpoint logging provides high-resolution visibility into QUIC behavior for development and debugging.
