TCP receive buffers are not what they appear

When a TCP sender pushes data into a socket, the receiving side is doing far more than simply storing bytes. The receive buffer budget, advertised window, and the kernel's internal memory accounting interact in ways that often surprise developers who assume a 64KiB buffer means 64KiB of data can arrive.

Two questions frame the investigation: how much data can actually sit in a TCP receive buffer, and how quickly can it get there? The answers depend as much on kernel metadata overhead and window-management heuristics as on the configured buffer size.

The buffer budget is split several ways

Linux divides the receive buffer budget, called sk_rcvbuf (reported as skmem_rb by ss), among several consumers:

  • Recv-q: application bytes waiting for read().
  • Metadata: the cost of struct sk_buff, struct skb_shared_info, and associated kernel structures.
  • Advertised window: the portion of remaining free space offered to the sender.

The first two together are reported as skmem_r (kernel name sk_rmem_alloc). What remains after that is nominally "free," but the kernel may divide it between the advertised window and future metadata needs. The tcp_adv_win_scale sysctl governs the upper bound: by default the window may occupy at most 50% of unclaimed space.

This budget does not behave like a simple bucket. When a socket is configured with setsockopt(SO_RCVBUF), the kernel doubles the supplied value for the actual budget, and autotuning is disabled. The default maximum for an unprivileged SO_RCVBUF is small — around 208KiB.

Advertising the whole buffer is a trap

Setting tcp_adv_win_scale=0, which advertises 100% of the buffer as window, appears attractive but breaks quickly. In a test with a 64KiB budget and full-size packets, the receive path exhausted its memory budget even though the advertised window still had bytes to offer. The result: skmem_r climbed past skmem_rb (73,984 versus 65,536 in one run), and nstat reported TcpExtTCPRcvQDrop increments — packets dropped despite an open window.

This is not a pathological corner case. A single struct sk_buff plus struct skb_shared_info can cost 576 bytes on common systems, often exceeding the payload itself. Network card allocation granularity can compound this: some drivers allocate a full page (4096 or even 16KiB) per packet regardless of payload size. Consequently, a packet with a small payload can consume far more socket memory than the bytes it carries.

When the budget is exhausted while the window remains open, the sender interprets the resulting drops as congestion and backs off. The data is eventually delivered, but exponential backoff can introduce serious latency spikes.

The kernel's defense: coalesce, collapse, and capping

Linux tries hard to avoid overflow without sacrificing performance. Two mechanisms operate on the receive queue:

  • TCP Coalesce: merges new data into the last struct sk_buff on the socket, retiring the per-packet metadata entirely. This runs constantly and is highly effective.
  • TCP Collapse: when the budget is already tight, the kernel rewrites and defragments the entire receive buffer into a few long segments, reducing metadata overhead. This is a heavier operation — one kernel developer has called frequent collapse "a major latency source."

For the collapse path to trigger, TCP must have overcommitted the memory budget despite the window cap. The rcv_ssthresh internal variable exists to keep that from happening. When collapse does run, rcv_ssthresh is lowered, effectively forcing a smaller advertised window until the kernel's cost model recalibrates.

Collapse becomes more likely with pathological packet patterns — for example, websocket traffic with many tiny frames and a slow reader. In such cases the nstat counters TcpExtTCPRcvCollapsed and TcpExtTCPRcvCoalesce are worth watching.

Window opening is gradual by design

Even in a clean run with default settings, the receive window does not spring open to its maximum. With a 64KiB budget and tcp_adv_win_scale=1, the window starts at 32KiB and slides as data arrives. Delayed ACKs produce a saw-tooth pattern as the receiver updates the window only when it processes ACKs.

For high-bandwidth, high-latency flows (large BDP), the ramp-up is more dramatic. A socket with a 2MiB budget does not advertise anything near 2MiB at first. rcv_ssthresh starts at 64KiB and grows by roughly two full-MSS packets for each packet whose truesize-to-payload ratio is favorable. The kernel is deliberately conservative: it wants to observe the actual per-packet cost on the wire before extending more credit, because a driver that allocates 16KiB buffers for 1500-byte segments will exhaust memory far faster than one that packs efficiently.

This slow start is fixed in vanilla kernels, but a patch allows raising the initial rcv_ssthresh via a per-route initrwnd option. With that in place, the first ACK can bump the window from the handshake-limited 64KiB up to 1MiB. The time to fill the buffer stays the same in terms of packets (~1800 in one test), but the sender injects data much earlier. That matters only for flows that combine long RTTs, large initcwnd, and immediate bulk data — a narrow set of use cases.

What this means in practice

Under default net.ipv4.tcp_rmem settings, Linux assigns a 128KiB initial budget. On a typical system with full-size packets and efficient coalescing, roughly 113KiB of application data can accumulate before the budget is consumed — the rest is metadata. Reaching that limit from an empty socket takes about six RTTs, with the sender injecting progressively smaller bursts as the window shrinks.

The same principles apply to QUIC, where receive buffer management is still maturing. The Linux TCP machinery — window caps, rcv_ssthresh ramp-up, and collapse avoidance — exists to balance throughput against memory and latency. The tradeoffs are invisible until they bite, usually in the form of unexplained drops or a sender that stalls because the receiver advertised credit it could not back with memory.