Unbounded memory usage by TCP for receive buffers, and how we fixed it

TCP receive buffers exceeding their limits

While monitoring our production systems, we noticed some TCP sessions were consuming far more memory than expected. The Linux kernel permits TCP sessions matching certain characteristics to bypass autotuning limits and allocate memory all the way up to net.ipv4.tcp_rmem max, the per-session cap. When enough of these sessions accumulate on a single server, total TCP memory hits net.ipv4.tcp_mem thresholds, and the kernel responds by restricting all TCP sessions. This causes TCP collapse processing, out-of-order (OFO) queue pruning, and dropped incoming packets — all of which degrade throughput and latency for every user on that box.

This article walks through the root cause and the fix we developed and tested.

Observing the problem

We began investigating after noticing a large number of TCP sessions with massive receive buffer allocations. Receive buffers hold packets that have arrived from the network but haven’t yet been read by the local process.

Most affected sessions had a round-trip time (RTT) of roughly 20ms. Standard bandwidth-delay product (BDP) math shows that a 2.5 MB window accommodates up to 1 Gbps at that latency. We counted sessions where autotuning’s upper memory limit (skmem_rb) exceeded 5 MB — double the calculated window. On one server, 558 sessions matched. Most looked like this:

Key fields:

• recvq — user payload bytes in the receive queue, waiting for the local process to read
• skmem "r" — actual kernel memory allocated for the receive buffer (sk_rmem_alloc)
• skmem "rb" — limit for r (sk_rcvbuf)
• l7read — user payload bytes read by the userspace process

The red flag is that both skmem_r and skmem_rb sat at 256 MiB — the system-wide maximum from net.ipv4.tcp_rmem. Autotuning should never let buffers grow that large for these sessions.

Plotting one session over time showed the problem clearly:

Every time skmem_r (allocated) exceeded skmem_rb (the limit), the limit was simply raised to match. Autotuning was being bypassed entirely.

Reproducing in the lab

Production traffic is too unpredictable for controlled experiments, so we ran extensive lab testing. After many attempts — and many dirty test machines — we narrowed it to a surprisingly minimal setup:

1. Sender: infinite loop, sending 1500-byte packets with a 1 ms delay between sends.
2. Receiver: infinite loop, reading 1 byte at a time with a 1 ms delay between reads.

That’s all it takes. The receive queue grows without bound until it hits net.ipv4.tcp_rmem max.

tcp_server_sender.py

import time
import socket
import errno

daemon_port = 2425
payload = b'a' * 1448

listen_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listen_sock.bind(('0.0.0.0', daemon_port))

# listen backlog
listen_sock.listen(32)
listen_sock.setblocking(True)

while True:
    mysock, _ = listen_sock.accept()
    mysock.setblocking(True)
    
    # do forever (until client disconnects)
    while True:
        try:
            mysock.send(payload)
            time.sleep(0.001)
        except Exception as e:
            print(e)
            mysock.close()
            break

tcp_client_receiver.py

import socket
import time

def do_read(bytes_to_read):
    total_bytes_read = 0
    while True:
        bytes_read = client_sock.recv(bytes_to_read)
        total_bytes_read += len(bytes_read)
        if total_bytes_read >= bytes_to_read:
            break

server_ip = “192.168.2.139”
server_port = 2425

client_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_sock.connect((server_ip, server_port))
client_sock.setblocking(True)

while True:
    do_read(1)
    time.sleep(0.001)

Confirming the failure

We ran the reproducer with:

• Kernel 6.1.14 vanilla
• net.ipv4.tcp_rmem max = 256 MiB (window scale factor 13)
• net.ipv4.tcp_adv_win_scale = -2

At second 189, this exchange occurred:

Memory limits are fully ignored here. When tcp_rmem max is reached:

• The kernel drops incoming packets.
• No ZeroWindow is ever sent — the receiver never tells the sender to stop.
• The sender retransmits with exponential backoff.
• Eventually (~15 minutes, depending on settings) the session times out with “Errno 110 Connection timed out.”

A range of packet sizes and send intervals triggers this. Our first reproduction was designed to grow the buffer quickly and doesn’t mirror production rates exactly.

Production traffic matched

Looking deeper at production streams, we found similar sessions — including one variant that initially looked different but shares the same root cause:

In that session, the application stopped reading entirely at second 411 (the L7read rate drops to zero). The bottom two graphs use log scales to show throughput and window size never actually reach zero.

This repeated packet pattern occurs during the growth phase after the reader stops:

That variant is covered later in the “Reader never reads” section.

Finding the code path

Since sk_rcvbuf is only modified in three places relevant here, we reviewed each:

• tcp_clamp_window
• tcp_rcv_space_adjust
• tcp_set_rcvlowat

We don’t call tcp_set_rcvlowat, so that was ruled out. Using bpftrace, we identified tcp_clamp_window as the culprit.

What we know so far

tcp_try_rmem_schedule runs as usual:

When rmem_alloc > sk_rcvbuf, tcp_try_rmem_schedule calls prune, which invokes tcp_clamp_window. Unexpectedly, tcp_clamp_window raises sk_rcvbuf to match rmem_alloc.

The core question: why does rmem_alloc ever exceed sk_rcvbuf?

TCP coalescing is the trigger

After more code review and bpftracing, the answer is TCP coalescing. This is distinct from Generic Receive Offload (GRO); it’s a TCP-level feature on the input path. When an incoming packet arrives, tcp_rcv_established calls tcp_queue_rcv, which invokes tcp_try_coalesce. If the payload can be appended to an existing packet, it is — saving header memory. Critically, rmem_alloc can rise above sk_rcvbuf due to the logic in that path.

Full failure chain

1. Data packets are received.
2. tcp_rcv_established coalesces, pushing rmem_alloc above sk_rcvbuf.
3. tcp_try_rmem_schedule → tcp_prune_queue → tcp_clamp_window raises sk_rcvbuf to match.
4. The kernel advertises a larger window based on the new sk_rcvbuf.

For step 2 to happen, rmem_alloc must already be near sk_rcvbuf. With tcp_adv_win_scale of -2, the window should be 25% of buffer size, so rmem_alloc shouldn’t approach sk_rcvbuf at all. In our tests, the truesize ratio wasn’t close to 4, so something else was off.

ZeroWindow never sent

ZeroWindow packets — advertising a window of zero — are how a receiver tells the sender to pause when its buffer is full. This mechanism should keep rmem_alloc well below sk_rcvbuf.

During testing we noticed the SNMP counter TCPWantZeroWindowAdv increasing while no ZeroWindow packets were actually sent. The window calculation logic was failing, and that led us to the root cause of all these symptoms.

When the Receive Window Doesn’t Actually Limit Anything

The TCP receive window, advertised in every ACK, tells the sender how much data the receiver is prepared to buffer. The math is simple: the ACK number plus the window size defines the right edge of the sender's allowed sequence space. But when window scaling is in use, there’s a granularity problem that breaks this accounting.

Window scaling exists because the original 16-bit window field couldn’t express large enough windows for modern high-bandwidth links. With scaling enabled, however, the receiver can only adjust the advertised window in discrete steps equal to the scale factor. Whenever the receiver acknowledges data that isn’t an exact multiple of that factor, it must move the right edge of the window — even if the actual buffer state hasn’t changed.

Most of the time, this is harmless. The right edge simply creeps to the right along with the ACK. Problems arise when the receive buffer approaches its configured limit. At that point, the receiver must either open the window beyond the limit or shrink it, because keeping the right edge stationary is impossible. Linux currently chooses to open the window, which is the equivalent of not having a limit at all. With any window scaling factor greater than one — which is to say, for virtually every TCP connection on the internet today — this behavior is in play.

Terminology and the Shrinking Question

TCP window management uses three terms that are often conflated:

• Closing the window — the left edge moves right as data is ACKed.
• Opening the window — the right edge moves right, increasing the advertised window.
• Shrinking the window — the right edge moves left, which is what RFC 7323 calls “retraction.”

Shrinking is distinct from simply advertising a smaller window size; it only occurs when the right edge itself moves backward.

Why Growing the Window is Wrong

When the right edge hits the upper bound set by receive buffer autotuning, there are only three paths forward:

1. Grow the window anyway
2. Drop incoming packets
3. Shrink the window

Growing the window is what Linux does today. It simply ignores the memory limit until the hard cap in net.ipv4.tcp_rmem is reached, at which point the kernel begins dropping packets. That behavior wastes memory, and when the cap is finally hit, the drop path causes the sender to retransmit with exponential backoff. The retransmitted packets are equally undeliverable, since userspace isn’t reading, and the session eventually breaks. It’s incorrect behavior for what is fundamentally a window-full condition.

The correct response is to shrink the window and send a ZeroWindow advertisement when the buffer is full. This costs no memory, no bandwidth, and doesn’t break connections.

RFC 7323 Already Answers This

The protocol specification anticipated the exact scenario. RFC 7323, section 2.4, states that a retracted window is permissible in certain instances and, more importantly, mandates that implementations handle a shrinking window:

The RFC’s Appendix F specifically addresses the case of a sender write that is smaller than the window scale factor, calling it a general problem. The Linux kernel, however, has not implemented the required handling — until now.

The Kernel Fix

The Cloudflare patch enabling TCP window shrinking has been merged upstream and will ship in Linux kernel 6.5 and later. The commit is available on GitHub.

With the patch, the repeated packet exchange pattern changes. Instead of growing beyond the limit and later dropping packets, the session now enforces the memory limit, sends ZeroWindows as needed, and avoids retransmissions entirely:

Testing at Realistic Scale Factors

Window Scale Factor of 8

A window scale factor of 8 with tcp_adv_win_scale set to 1 is common on the public internet. Testing this configuration on kernel 6.1.14 vanilla with tcp_rmem max at 8 MiB produced the same failures as the earlier wscale-13 test — memory runaway and packet drops beginning around the 2100-second mark. With the patch applied, the session behaved correctly.

Oscillating Reader

In a test where the reader alternates every 240 seconds between slow reads (1 byte per millisecond) and fast reads (3300 bytes per millisecond), with a 256 MiB tcp_rmem max and tcp_adv_win_scale at −2, the vanilla kernel let the receive buffer grow across the entire run. The patched kernel kept skmem_rb to roughly 20 MB. That value might not be optimal for the session, but it is bounded rather than monotonically increasing.

Reader Never Reads

The extreme case is a reader that never calls read() at all. With the sender transmitting 4 bytes every millisecond and a 8 MiB tcp_rmem max, the unpatched kernel allowed the receive queue to fill to the full 8 MiB. The patched kernel paused the flow after only a few packets, as would be expected.

Production Results at Cloudflare

Deployed across Cloudflare’s production network, the patch’s impact is visible in aggregate. Rollout on most servers occurred May 1 at 22:00.

Eliminated Packet Drops and Collapse Processing

The RcvPruned metric, counting incoming packets dropped due to memory constraints, went to zero. Similarly, TCPRcvCollapsed — packets merged to reclaim memory by eliminating header metadata — dropped to zero across all servers, along with the CPU time spent in collapse processing. That latency previously competed with HTTP request processing.

Memory Reduction

With autotuning limits genuinely enforced, total TCP buffer memory allocation decreased. Comparing the same data center seven days apart — before and after the patch — shows a clear reduction in memory footprint.

ZeroWindows Are Sent as Intended

The TCPWantZeroWindowAdv counter tracks the number of times the window calculation indicated a ZeroWindow should be sent but wasn’t. After the patch, this counter stops, meaning the advertised window now correctly reflects the memory constraint. The related TCPZeroWindowDrop metric — packets dropped while the session is in a ZeroWindow state — remains nonzero, but these drops are harmless. They are the result of data packets and the ZeroWindow advertisement crossing on the wire, and they have no negative impact because the sender is already blocked.

Importantly, Cloudflare has not observed any peer TCP stack that fails to handle the shrinking window, indicating that the patch does not introduce interoperability problems with RFC-compliant implementations. The patch enforces existing memory limits, eliminates unnecessary drops and collapse processing, and brings Linux’s receiver behavior into line with the specification.