Cloudflare's speed test site, speed.cloudflare.com, launched in May 2020 with download, upload and latency measurements. A small number of users soon reported that upload results sometimes came out too low, and the pattern pointed at connections with several hundred Mbps of available upstream bandwidth — cable or fiber class. Since the tests run as browser JavaScript and most browsers default to HTTP/2, the suspicion fell on the protocol itself: under high upload bandwidth, HTTP/2 transfers were sometimes far slower than HTTP/1.1 over TLS.

The stakes go beyond a test page. Pandemic-era home connections are carrying more upstream traffic than before — video conferencing, content creation, working from a home office — after years of highly asymmetric download-heavy use.

Reproducing the gap in a lab

To work in a controlled setting, upload tests were run inside a Linux VM on a MacBook Pro, with dummynet placed between the VM and the Mac host. The methodology was deliberately simple: create a file, upload it with curl to an endpoint that accepts a request body, repeat 20 times and take the median speed in Mbps.

% dd if=/dev/urandom of=test.dat bs=1M count=10
% curl --http1.1 -w '%{speed_upload}\n' -sf -o/dev/null --data-binary @test.dat https://edge/upload-endpoint
% curl --http2 -w '%{speed_upload}\n' -sf -o/dev/null --data-binary @test.dat https://edge/upload-endpoint

At 200Mbps of available bandwidth and 40ms RTT, a 10MB upload using production configuration ran at almost half the speed of the same conditions over HTTP/1.1. The slower the link, the smaller the gap: on a cable connection with 5Mbps upload and 20ms RTT, the two protocols performed almost identically. The problem appears on fast networks.

Delivering HTTP/2 upload speed improvements Embedded Image - gszaRn

Why receiver flow control is the suspect

When a client uploads, the server becomes the receiver of data, and the receiver needs a way to manage its receive buffer. HTTP/1.1 has no protocol-level mechanism for this: without multiplexed requests it leaves flow control to TCP, and modern OS stacks autotune the TCP receive buffer against the bandwidth-delay product (BDP).

HTTP/2 does define stream-level flow control because streams are multiplexed, with a window per stream plus a connection-wide window. Too tight a window blocks the sender; too loose wastes memory on buffering. The best strategy keeps the receive buffer matched to the current BDP, which is the maximum number of bytes in flight on the link.

Inside NGINX request body buffering

The first attempt was to tune configuration. Cloudflare does not use the proxy_request_buffering directive, which rules it out. The relevant settings are:

  • proxy_buffering
  • client_body_buffer_size — the request body buffer size, shared by HTTP/1.1 and HTTP/2
  • http2_body_preread_size — HTTP/2 only
Delivering HTTP/2 upload speed improvements Embedded Image - DzQuG1

The HTTP/1.1 path uses the client_body_buffer_size buffer to shuttle data between client and upstream, alternating reads and writes, while TCP socket buffers do flow control. The HTTP/2 path interacts with flow control windows and works like this:

  • A request body buffer is allocated at the larger of http2_body_preread_size and client_body_buffer_size. With 64KB for the former and 128KB for the latter, the allocation is 128KB. Cloudflare uses 128KB as the client_body_buffer_size.
  • The stream's INITIAL_WINDOW_SIZE is set to http2_body_preread_size, defaulting to 64KB (the RFC7540 default).
  • The HTTP/2 module reads up to http2_body_preread_size before passing data upstream.
  • After the preread buffer is flushed, it keeps reading from the client, writing upstream and returning WINDOW_UPDATE frames as needed until the body is complete.

In short, HTTP/2 adds application-layer receiver flow control on top of TCP, and NGINX's receiver uses a fixed-size buffer. Whenever the link's BDP exceeds that size, the flow control window becomes the bottleneck.

Sizing the buffer by hand

Larger buffers ought to remove the bottleneck, so the original result was plotted as "prod" against tests using 256KB, 512KB and 1024KB for client_body_buffer_size.

Delivering HTTP/2 upload speed improvements Embedded Image - rMiTXl

At 40ms RTT, 512KB appeared optimal. At 10ms RTT, 256KB appeared optimal.

Delivering HTTP/2 upload speed improvements Embedded Image - OyXtm6

Both cases came close to or beat HTTP/1.1, but the ideal size moves with the network, and an oversized buffer hurts performance. Static tuning cannot track that, so the buffer needs to size itself.

Autotuning the receive buffer

TCP stacks already solve this problem by autotuning the receive buffer, and Cloudflare's edge runs with TCP receiver buffer autotuning enabled by default.

net.ipv4.tcp_moderate_rcvbuf = 1

On its own, TCP autotuning does not help here: the HTTP/2 layer enforces its own flow control, and the fixed 128KB was simply too small for high-BDP links. The fix was to autotune the HTTP/2 receive buffer in much the same way TCP does. The algorithm in Cloudflare's NGINX builds doubles the HTTP/2 request body buffer size according to the measured BDP:

  • Allocate a request body buffer as above.
  • Once per RTT, using Linux tcp_info, update the current BDP.
  • Double the buffer size when the current BDP exceeds a quarter of the receiver window.

Results

Lab

With HTTP/2 autotune enabled and client_body_buffer_size still at 128KB, the "h2 autotune" run performed close to HTTP/1.1 and at times slightly faster, which was the initial target. A hand-picked optimal size can still beat it in a given scenario, but NGINX now converges on an appropriate buffer size on its own as conditions change.

Delivering HTTP/2 upload speed improvements Embedded Image - k9gR7I
Delivering HTTP/2 upload speed improvements Embedded Image - pWvouA

Production

The follow-up test was run against Cloudflare's production edge, uploading a 10MB file from a Google Cloud Linux VM over a multi-Gbps, sub-10ms link. From Google Cloud's Belgium region to the Paris (CDG) PoP at 7ms RTT, the improvement was close to 3x.

Delivering HTTP/2 upload speed improvements Embedded Image - zdWqCe

Tokyo-to-Tokyo (NRT PoP) at 2.3ms RTT is not representative of home users, but the numbers are still informative: the fixed 128KB buffer performs well there, and HTTP/2 with autotune outperforms HTTP/1.1.

Delivering HTTP/2 upload speed improvements Embedded Image - osJ3Nk

Outcome

HTTP/2 upload buffer autotuning is now deployed across the Cloudflare edge, so all HTTP/2 connections — including tests on speed.cloudflare.com — should see the faster upload path. Download speed and latency dominate performance discussions, but upstream throughput matters for photo and video sharing, content creation, video conferencing and broadcasting from home. The patch has also been released publicly and submitted to the NGINX upstream project.