Optimizing TCP for high WAN throughput while preserving low latency

When Low Latency and High Throughput Collide

TCP performance tuning is rarely a single-variable problem. Cloudflare's own history illustrates this: in 2015, engineers discovered latency spikes in HTTP request processing caused by the kernel spending too much time collapsing receive queues. The fix was to set tcp_rmem to 4 MiB, minimizing kernel collapse work. That worked well for terminating connections at edge servers close to users.

But new products—Magic WAN, WARP, Spectrum, Gateway—changed the traffic profile. Office-to-office traffic over Magic WAN carries large file transfers (SMB, for example): elephant flows across long fat networks where throughput dominates the user experience. The old tradeoff no longer holds. Cloudflare now needs both low latency and high throughput on high-latency paths.

Key Players in Receive Buffer Management

Understanding the constraint requires knowing how several Linux TCP mechanisms interact:

• TCP receive window — The maximum unacknowledged user payload bytes a sender may transmit. It fluctuates over a session and is the usual limiter on high-latency throughput.
• net.ipv4.tcp_adv_win_scale — A counterintuitive multiplier accounting for packet-processing overhead. The receive window is expressed in user bytes; the kernel needs additional memory for metadata. The scale factor determines how the maximum window derives from available buffer space per the table below.
• sk_rcvbuf — Per-socket ceiling on receive buffer memory. Settable via SO_RCVBUF, though generally discouraged except for localhost sessions; proper values depend on session latency and other dynamics that applications can't know in advance.

How Autotuning Works

Linux autotuning resolves the application's dilemma by adjusting buffer limits dynamically. The kernel tracks the local application's read rate from the receive queue and the session RTT, then raises buffers and receive window until the application or the network becomes the bottleneck—or limits them when the local reader is slow, preventing bufferbloat.

The relevant state is visible via ss -tmi:

• Recv-Q — User payload bytes not yet read by the application.
• rcv_ssthresh — The receiver-side clamp on the receive window. The sender only sees the current window; this value stays local.
• skmem_r — Actual allocated memory, including packet metadata beyond user payload (sk_rmem_alloc).
• skmem_rb — The socket's maximum allocatable receive memory (sk_rcvbuf), raised by autotuning up to tcp_rmem max.
• rcv_space — High-water mark of the application's read rate within one RTT; used internally to adjust sk_rcvbuf.

net.ipv4.tcp_rmem (third value) is the global failsafe capping how high autotuning can push a socket's receive buffer. Under normal conditions it plays a minor role. Receive buffer memory is not preallocated; it's allocated as packets arrive. And crucially, autotuning does not wait for a full receive queue to expand buffers—preventing excessive buffering is one of its benefits.

Why Large Windows Are Non-Negotiable

High bandwidth-delay product (BDP) sessions require large receive windows. There's no way around the physics: fiber latency between Cloudflare's Zurich and Sydney facilities measures about 300 ms, and a reasonable worst-case throughput target is 3500 Mbps for a single session on modern hardware. That yields a BDP of about 131 MB, rounded to 128 MiB. Linux autotuning handles lower-latency and constrained sessions correctly on its own, so only the maximum matters.

The receive buffer must be larger than the window to accommodate packet metadata. On some Cloudflare hardware with full-sized packets, measured allocations reach three times the payload size. To reduce TCP collapse frequency, tcp_adv_win_scale is set to -2, making the maximum window one quarter of the buffer space. That produces the following sysctl configuration:

net.ipv4.tcp_rmem = 8192 262144 536870912
net.ipv4.tcp_wmem = 4096 16384 536870912
net.ipv4.tcp_adv_win_scale = -2

With tcp_rmem at 512 MiB and tcp_adv_win_scale at -2, autotuning can set a maximum window of 128 MiB—exactly the design target.

The Case Against TCP Collapse

When a full receive buffer gets a new packet, Linux hasn't simply dropped it. It first attempts to collapse the queue—a memory defragmentation step that is not guaranteed to succeed and costs CPU time, which showed up as the original latency spikes. The collapse path triggers when a socket fills its receive queue: autotuning opened the window for a fast reader, then the application slowed down.

Dropping the packet outright creates no problems. The receive queue is already full, the local application still has data, and the sender's congestion control or ZeroWindow handling responds correctly.

Linux provides no sysctl to disable TCP collapse. Cloudflare therefore developed an in-house kernel patch to skip the collapse logic entirely.

Patch Attempt #1: A Partial Success

The first kernel patch was simple: at the top of tcp_try_rmem_schedule(), return immediately on memory allocation failure (after clearing pred_flag and resetting SACK), bypassing tcp_collapse and related code entirely.

The latency spikes disappeared. But expected throughput did not materialize.

The investigation revealed a testing blind spot. Standard benchmarking tools like iperf3 never fill the receive queue, so autotuning never opens the window wide enough to expose the target behavior. Autotuning behaved correctly for such well-behaved readers. Cloudflare needed application-layer software that stresses the autotuning logic—a less well-behaved consumer that reads data at one rate, then slows down—to properly exercise and evaluate the patched kernel.

Building a better TCP benchmark

The anomalies uncovered during the first patch attempt only appeared under specific conditions, which made them hard to detect with standard tools. To measure their impact, we built a dedicated benchmarking suite consisting of two Python programs.

The reader program opens a TCP session to the daemon, which immediately begins sending user payload at maximum speed and never stops. The reader then alternates between opening the receive window wide and forcing the buffers to fill completely:

1. Reads at full speed for five seconds (fast mode) to open the window
2. Calculates 5% of the high watermark of bytes read during any previous one-second interval
3. For each of the next 15 seconds (slow mode):
   • Reads that 5% number of bytes, then stops
   • Sleeps for the rest of the second
4. Repeats steps 1–3 three times, for a total run of 60 seconds

This pattern repeatedly drives the buffers to their limits, exposing any packet-handling issues that only surface under sustained pressure combined with full receive buffers.

Why default Linux collapse behavior doesn't work

On kernel v5.15.16, Linux handles receive buffer exhaustion by aggressively reclaiming memory to make room for incoming packets. While this avoids drops, it comes at the cost of latency:

With the default tcp_prune_queue() path, latency spikes in tcp_try_rmem_schedule() reach:

@ms:
[0]       27093 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[1]           0 |
[2, 4)        0 |
[4, 8)        0 |
[8, 16)       0 |
[16, 32)      0 |
[32, 64)     16 |

With tcp_rmem 146 MiB and tcp_adv_win_scale +3:

@ms:
(..., 16)  25984 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[16, 20)       0 |
[20, 24)       0 |
[24, 28)       0 |
[28, 32)       0 |
[32, 36)       0 |
[36, 40)       0 |
[40, 44)       1 |
[44, 48)       6 |
[48, 52)       6 |
[52, 56)       3 |

With tcp_rmem 137 MiB and tcp_adv_win_scale +4:

@ms:
(..., 16)  37222 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[16, 20)       0 |
[20, 24)       0 |
[24, 28)       0 |
[28, 32)       0 |
[32, 36)       0 |
[36, 40)       1 |
[40, 44)       8 |
[44, 48)       2 |

These spikes are measured in milliseconds — far too high for the latency requirements of a global production network.

Patch attempt #2: guarding the collapse

The first attempt failed because the receive queue memory limit was hit early in the flow's ramp-up, when sk_rmem_alloc and sk_rcvbuf were still around 800KB. This happened at approximately the two-second mark for the 137p4 configuration (about 2.25 seconds for 170p2).

The key insight was that tcp_prune_queue() itself raises sk_rcvbuf when it can. So we revised the patch to permit collapse only when sk_rmem_alloc is below a configurable threshold:

net.ipv4.tcp_collapse_max_bytes = 6291456

The updated patch is available here. Results with the new guard in place:

oscil – 300ms tests

oscil – 20ms tests

oscil – 0ms tests

iperf3 – 300ms tests

iperf3 – 20ms tests

iperf3 – 0ms tests

All tests pass.

Choosing tcp_collapse_max_bytes

The setting must reflect the largest queue we can collapse while still keeping latency acceptable.

A 6 MiB threshold keeps maximum collapse-induced latency under 2 ms.

Production validation

Current production settings

net.ipv4.tcp_rmem = 8192 2097152 16777216
net.ipv4.tcp_wmem = 4096 16384 33554432
net.ipv4.tcp_adv_win_scale = -2
net.ipv4.tcp_collapse_max_bytes = 0
net.ipv4.tcp_notsent_lowat = 4294967295

With tcp_collapse_max_bytes set to 0, the custom logic is disabled and vanilla kernel collapse processing is used.

New settings under test

net.ipv4.tcp_rmem = 8192 262144 536870912
net.ipv4.tcp_wmem = 4096 16384 536870912
net.ipv4.tcp_adv_win_scale = -2
net.ipv4.tcp_collapse_max_bytes = 6291456
net.ipv4.tcp_notsent_lowat = 131072

Note that tcp_notsent_lowat is covered below, and the middle tcp_rmem value was changed due to separate work showing Linux autotuning was sizing receive buffers too large for localhost sessions. That change reduces TCP memory usage for local traffic but doesn't affect the long-haul sessions relevant here.

Benchmarks used non-Cloudflare hosts in Iowa, US and Melbourne, Australia transferring data to the Cloudflare data center in Marseille, France. Marseille hosts ran either current production settings or the new configuration, using perf3 version 3.9 on kernel 5.15.32.

Throughput results

Iowa-Marseille throughput

Iowa-Marseille receive window and bytes-in-flight

Melbourne-Marseille throughput

Melbourne-Marseille receive window and bytes-in-flight

The Melbourne-to-Marseille path remains limited by the receive window on the Cloudflare host even with the new settings, meaning further tuning can unlock additional throughput.

Latency results

The Y-axis in these charts is the 99th percentile TCP collapse time in seconds.

Cloudflare hosts in Marseille running current production settings

Cloudflare hosts in Marseille running the new settings

Maximum collapse time with the new settings is no worse than production — exactly the outcome we wanted.

Sender-side: tcp_notsent_lowat

The receiver is only half the story. With tcp_wmem max set to 512 MiB, oscillating reader flows can inflate the send buffer significantly — bufferbloat and wasted kernel memory.

tcp_notsent_lowat provides the fix by capping unsent bytes in the write queue:

With an RTT of 466ms, throughput remains at full wire speed (1 Gbps) in all cases. TCP memory usage, as reported by /proc/net/sockstat, drops dramatically.

Our web servers already set tcp_notsent_lowat to 131072 per socket. Other senders use the kernel default of 4 GiB. Changing the sysctl applies the 131072 limit to all senders on the server.

Summary

The combined changes open the throughput floodgates for high-BDP connections while keeping HTTP request latency minimal — demonstrated both in controlled tests and across real cross-continental paths in production.