When "idle" isn't idle: how a Linux kernel optimization became a QUIC bug

A congestion collapse that never ends

CUBIC, standardized in RFC 9438, is the default congestion controller in Linux and governs how most TCP and QUIC connections on the public Internet probe for bandwidth, back off on loss, and recover. At Cloudflare, our open-source QUIC implementation, quiche, uses CUBIC as its default, putting this code in the critical path for a significant share of the traffic we serve. This is the story of a bug where CUBIC's congestion window (cwnd) gets pinned permanently at its minimum and never recovers from a congestion collapse event.

The root cause traces back to a Linux kernel change that aligned CUBIC with the app-limited exclusion described in RFC 9438 §4.2-12. That fix addressed a real problem in TCP — but when ported to quiche, it surfaced unexpected behaviors. The resolution is an elegant, near-one-line fix.

The CUBIC state machine

Congestion Control Algorithms (CCAs) turn one central knob: the congestion window (cwnd), the sender-side cap on bytes in flight. A larger cwnd pushes more data per round trip; a smaller one throttles the sender. Every loss-based CCA, CUBIC included, is ultimately a policy for growing cwnd when the network looks healthy and shrinking it when it doesn't.

The underlying premise of loss-based algorithms like CUBIC is straightforward:

1. If there is no packet loss, increase the sending rate to raise bandwidth utilization.
2. If there is loss, assume the network's capacity has been exceeded and back off.

This logic rests on assumptions that have been revisited over the years, but those are topics for a separate discussion.

The symptom: a test that fails 61% of the time

The investigation began with unexpected failures in our ingress proxy integration test pipeline. The erratic behavior appeared in tests where CUBIC was evaluated under heavy loss in the early part of the connection.

Recovery after congestion collapse is an uncommon regime — but it is exactly what a congestion controller exists to handle. Most congestion control tests exercise steady-state and growth phases; few probe behavior at minimum cwnd, after the connection has been beaten down. Bugs in this corner of the state space are invisible in throughput dashboards, undetectable by static review, and surface only when you deliberately drive a CCA into it and watch whether it can climb back out. This test did exactly that.

The simulated test setup included:

• A quiche HTTP/3 client and server running locally (localhost)
• RTT = 10ms (set in the configuration)
• A 10 MB file download over HTTP/3
• CUBIC congestion control
• 30% random packet loss injected during the first two seconds, then nothing
• A generous 10-second timeout for a download expected to complete in four or five seconds

Expected behavior: CUBIC takes hits during the loss phase, reduces its congestion window, and once loss stops, ramps up steadily and finishes well within the timeout. Instead, in multiple 100-time runs, around 60% of tests failed to complete the download within 10 seconds.

The anomaly: 999 state transitions with zero loss

We instrumented quiche's qlog output with packet loss events and built visualizations of the congestion controller's behavior:

After the two-second mark, packet loss stops entirely. Yet the number of bytes in flight remains flat, contradicting CUBIC's core logic: in the absence of loss, apply more throttle. If the network is no longer dropping packets, why is the congestion window failing to grow?

Zooming into that region reveals CUBIC entering a rapid oscillation between congestion avoidance (the operational regime) and recovery (the packet loss recovery state) — 999 transitions in approximately 6.7 seconds, or one transition every ~14ms. That period is suspiciously close to the connection's RTT of 10ms. Throughout the entire period, cwnd is locked at the minimum floor: 2700 bytes, or two full-size packets.

Something in CUBIC's logic is misinterpreting the connection's state. The key clue is the oscillation period: ~14ms matches the RTT. Whatever triggers the recovery/avoidance flip happens once per round trip, in lockstep with the ACK clock — the self-clocking rhythm where each round trip's ACKs from the client trigger the server's next send. In this download scenario, ACKs travel client to server, and CUBIC's state machine runs on the server side. Every time those ACKs land, bytes_in_flight drops to zero and the server sends the next two-packet burst — which is exactly what triggers the bug.

To confirm the behavior was CUBIC-specific, we ran the same test with Reno, another loss-based algorithm with a different growth rate. The result: 100% pass rate. Reno recovered cleanly after the loss phase, confirming this was a CUBIC-related bug.

Why CUBIC misreads idle time

CUBIC's growth function is anchored to an epoch — a reference timestamp reset when the algorithm restarts its growth curve, most importantly after a loss reduces cwnd. Between resets, the curve parameter delta_t = now - epoch_start grows with wall-clock time. If an application idles for a long stretch and then resumes, delta_t becomes enormous, and CUBIC would attempt to inflate the window to an absurd value.

A 2017 kernel fix addressed this by shifting the epoch forward by the idle duration rather than resetting it to the current time. That preserved the curve's shape — sliding it later in time so growth picks up from where it left off, instead of restarting the steep post-loss climb.

Porting to quiche introduced a flaw

When CUBIC was ported to quiche in 2020, the idle adjustment came along — but QUIC runs in user space and lacks TCP's kernel-level CA_EVENT_TX_START callback. Quiche instead checks for idle inside on_packet_sent():

// cubic.rs — on_packet_sent() (simplified)
/// Updates the state when a packet is sent.
fn on_packet_sent(&mut self, bytes_in_flight: usize, now: Instant, ...) {
    // If the sending burst is restarting (i.e., bytes_in_flight was zero before this send),
    // adjust the congestion recovery start time to account for the gap in sending.
    if bytes_in_flight == 0 {
        let delta = now - self.last_sent_time;
        self.congestion_recovery_start_time += delta;
    }
    // Record the time of this send event.
    self.last_sent_time = now;
}

The port included a bug from the original kernel change, fixed about a week later by a follow-up to tcp_cubic. That kernel fix noted that setting epoch_start based on send-time tracking is imprecise, because the value is normally set during ACK processing. The correction: never set epoch_start in the future. If the recovery start time is pushed ahead of the current ACK time, bictcp_update() can overflow and CUBIC again grows cwnd too aggressively.

The quiche implementation computed the idle adjustment from send timestamps, which could shove the recovery boundary into the future. That only happens consistently when every incoming ACK drains bytes_in_flight to zero — meaning cwnd is pinned at its two-packet floor and the application has a full window ready the moment an ACK lands. Outside that regime, bytes_in_flight == 0 rarely holds on every send, so the trap stays dormant.

Why connection start is safe

The bug can't fire during slow start. Before the connection exits slow start, congestion_recovery_start_time is unset, so the buggy branch has no recovery boundary to advance. During slow start, CUBIC grows cwnd with the same ACK-counting rule Reno uses; the cubic curve only matters once congestion avoidance begins. The trap needs three conditions simultaneously: a real loss event to set the recovery boundary, congestion avoidance active, and the window collapsed to two packets.

The self-perpetuating recovery loop

At minimum cwnd, the connection enters a cycle where the idle optimization becomes self-fulfilling:

1. Send and ACK: The sender transmits the full two-packet window. One RTT (~14ms) later, both packets are ACKed and bytes_in_flight drops to zero.
2. False idle: The next send sees bytes_in_flight == 0 and assumes idle — but the pipe was congestion-limited, not idle.
3. Inflated delta: The idle duration is computed as now - last_sent_time. At minimum cwnd, last_sent_time is the start of the previous RTT cycle, so the delta is roughly the full RTT — not the actual processing gap, which is near zero. The recovery start time shifts forward aggressively, potentially into the future.
4. Perceived recovery: With the recovery start time in the future, in_congestion_recovery() returns true on every ACK. Processing the ACK exits recovery and sets the start time to the ACK time — which is later than last_sent_time, making the next send likely to push the boundary forward again.
5. Stagnation: CUBIC skips cwnd growth during recovery, so the window stays at two packets. The pipe drains completely on the next ACK, and the cycle restarts.

The loop repeats for thousands of cycles, breaking only when scheduler jitter and ACK-processing variance accumulate enough for the <= boundary in in_congestion_recovery() to fall behind the next send time.

Measuring idle from the last ACK

The fix redefines the idle measurement: instead of tracking from the last packet sent, the duration starts when bytes_in_flight actually transitioned to zero — the moment the last ACK was processed.

1. Add a last_ack_time timestamp to the CUBIC state.
2. Update it when ACKs arrive.
3. Use it in the idle delta computation.

// cubic.rs — on_packet_sent()
fn on_packet_sent(&mut self, bytes_in_flight: usize, now: Instant, ...) {
    // Check if the connection was idle before this packet was sent.
    if bytes_in_flight == 0 {
        if let Some(recovery_start_time) = r.congestion_recovery_start_time {
            // Measure idle from the most recent activity: either the
            // last ACK (approximating when bif hit 0) or the last data
            // send, whichever is later. Using last_sent_time alone
            // would inflate the delta by a full RTT when cwnd is small
            // and bif transiently hits 0 between ACK and send.
            let idle_start = cmp::max(cubic.last_ack_time, cubic.last_sent_time);

            if let Some(idle_start) = idle_start {
                if idle_start < now {
                    let delta = now - idle_start;
                    r.congestion_recovery_start_time =
                        Some(recovery_start_time + delta);
                }
            }
        }
}

With the delta reflecting the real gap since the last ACK, the recovery boundary stops chasing the send time:

For genuinely idle connections, last_ack_time sits far in the past, so the same expression still captures the full idle duration and the original epoch-shift behavior is preserved.

Validation and takeaways

With the fix in place, the quiche test suite returned to a 100% pass rate. The congestion window grows along the expected CUBIC curve, and the download completes in roughly 4–5 seconds.

The losses near the end of the connection are expected — they reflect full utilization of the router's allocated buffer.

• "Idle" resists simple definition. Normal pipeline delays at small windows can look like idleness to naive checks.
• Minimum-cwnd dynamics are a distinct corner case. The bug stayed invisible at high throughput and only surfaced after severe loss collapsed the window.
• The fix was tiny relative to the behavior's complexity. Weeks of qlog instrumentation and visualization analysis led to a change of roughly three lines.

The fix has been contributed to cloudflare/quiche, Cloudflare's open-source QUIC and HTTP/3 implementation. Beyond loss-based algorithms, quiche's modular congestion control design is also used to experiment with the model-based BBRv3 implementation, now enabled for a growing share of Cloudflare's QUIC deployments.