A Race Condition Hidden in hyper's HTTP/1 Loop

Cloudflare's Images service, built in Rust on Workers, runs on every machine at the edge. When the team rearchitected the Images binding in December 2025 to provide a more direct, local connection between the Workers runtime and the Images service, they expected faster performance. Instead, they got intermittent failures for larger image transformations — responses that returned HTTP 200 with no logged errors, but with bodies cut short by megabytes.

What followed was six weeks of investigation into a race condition in the open-source hyper HTTP library. The fix ultimately required only a few lines of code.

The Binding Architecture

Bindings give Workers direct APIs to platform resources. The Images binding decouples image optimization from delivery, letting developers transcode, composite, or manipulate images programmatically without returning the output as an HTTP response. A worker can pass image data to the Images API, chain operations, and receive the processed result as a stream.

Data flows from the Workers runtime through an intermediary service to the Images service over a socket. The socket's buffers, managed by the operating system kernel, hold data temporarily after one side writes it and before the other reads it. Hyper manages the connection on the Images service side, reading requests and writing responses.

Originally, the binding routed through FL, an internal intermediary service handling security and performance features. Each change to the binding had to follow FL's release cycle. In December 2025, the team replaced FL with an internal worker binding on the same machine, using Unix sockets to bypass FL's network stack overhead. Within days, the first bug report arrived.

200 OK, But Not Really

The affected customer used a nested setup: an inner pipeline using the Images binding to composite multiple large JPEG and PNG sources from R2 into one image, then an outer pipeline using the URL interface to compress, transcode, and resize the result. The outer pipeline received HTTP 200 from the inner one, with a Content-Length header promising several megabytes. The actual body was much smaller — in one request, only ~200 KB of 3.3 MB arrived.

Debugging moved inward through the request path, layer by layer:

  • Reproduction: A worker mimicking the customer's nested setup triggered the bug with the binding alone — in one run, 19 of 25 requests failed. The ~200 KB that arrived matched the production socket buffer size.
  • Timeouts ruled out: Truncation did not correlate with request duration.
  • Hyper versions tested: The bug appeared in hyper 0.14, 1.7, and 1.8 — no upstream fix existed.
  • Local reproduction failed: Tests on macOS and a Debian VM, direct curl requests, and request replays never failed. The bug required real concurrency with a Workers runtime client.
  • Workers runtime cleared: Traces from both sides of the connection showed no unexpected closes, and other services used the same client without issues.
  • Distributed tracing: The truncated body was already present before the outer pipeline, narrowing the problem to the binding path through Images.
  • Intermediary service ruled out: Instrumentation showed bodies were already truncated when they left the Images service.

The consistent signal: the bug was timing-dependent, appearing only in production with real concurrency and larger images.

What the Kernel Saw

Application-level tracing reported everything was fine — responses sent, no errors, 200 on every request. The team attached strace to the Images service to record actual syscalls. Setup was delicate: strace adds timing overhead per intercepted syscall. A narrow syscall filter kept that to a minimum, but broadening it shifted the timing enough to make the bug disappear entirely — reinforcing the timing-sensitivity theory.

Comparing syscall output between successful and failing requests revealed the problem. In a successful request, the response was written in chunks as the socket buffer allowed, with shutdown called only after all data was sent. A failing request showed a single write of headers plus a sliver of the body, followed immediately by shutdown. Out of a 14.9 MB response, only ~219 KB was sent. The remaining data never left hyper's internal buffer, and no termination signal came from the client. The Images service shut down the connection on its own, believing it was done.

The December rearchitecture didn't create this bug — it had existed in hyper for years across multiple major versions. But the new intermediary read at a pace that occasionally let the socket buffer fill during larger responses. FL presumably consumed data fast enough to avoid this. A few milliseconds of backpressure from an improvement that made everything faster surfaced the flaw.

The Missing Flush Check

Hyper's HTTP/1 connection lifecycle is driven by a state machine in dispatch.rs. The loop reads requests, writes responses, flushes the write buffer, and decides when to shut down. The bug lived in this line:

let _ = poll_flush(...)

In Rust, let _ = expr discards the expression's result — including Poll::Pending, the signal that the flush isn't done. When a request failed, the sequence was:

  1. The Images service hands the entire encoded response to hyper as a single in-memory block.
  2. Hyper writes the block to its internal buffer and marks its write state as Writing::Closed — encoding is done.
  3. Hyper calls poll_flush to move buffered data to the socket. The socket accepts ~219 KB; the remaining ~14.8 MB stays in the buffer. The full socket returns Poll::Pending.
  4. poll_loop discards the pending signal with let _.
  5. wants_read_again() returns false — the full request was already received.
  6. poll_loop returns Poll::Ready(Ok(())), signaling completion even though the flush isn't done.
  7. poll_shutdown() issues SHUT_WR.
  8. The client receives 219 KB plus EOF, though it expects 14.9 MB.

Hyper marked the write as complete when the response was buffered, not when it was flushed. When the socket buffer was full, the flush had to wait — but hyper didn't, proceeding to shut down with data still in its buffer.

This explains why curl never triggered the bug: curl reads as fast as data arrives, so the socket buffer never fills and the flush completes immediately. The production reader occasionally paused for a few milliseconds, filling the buffer at the wrong moment.

The Fix

The team built a deterministic test using a custom wrapper around a TCP stream that simulated a full socket buffer — accepting 8 KB on first write, then returning Poll::Pending on subsequent writes. Sending a 500 KB response through this constrained socket confirmed that hyper called shutdown with 492 KB still buffered.

The initial fix modified the dispatch loop to check whether the flush was actually done instead of discarding its result, returning Poll::Pending to the async runtime if not. This worked — every byte was written, and shutdown only followed an empty buffer. But the dispatch loop wasn't the right place: early Poll::Pending could reduce read polling frequency and slow other operations on the connection, and it didn't correctly handle keepalive connections, which should remain reusable while a previous response is still flushing.

The more targeted fix applies at the point where shutdown is called. Before shutting down the socket, hyper first flushes any remaining data in its buffer. This leaves the dispatch loop unchanged, adding a flush only at the exact moment before shutdown where data loss would otherwise occur.

The fix and deterministic test were merged into hyperium/hyper via PR #4018 and will appear in a future hyper release. Cloudflare is running an internal fork with the patch applied.

Lessons from the Kernel

Application-level observability never surfaced errors, crashes, or useful log entries. The failure was intermittent, scaled with response size, resisted reproduction with curl, and vanished under closer observation — classic symptoms of a timing-dependent bug in the connection layer, not application logic.

The breakthrough came from kernel-level tooling with strace, which recorded what actually happened on the socket. The bug lived in the few milliseconds between a partial flush and premature shutdown — a window that only opened after the system was made faster.