Why we killed keepalives between two local services

Keepalives are usually a no-brainer: reusing a TCP connection avoids handshake overhead and speeds up requests. But when two services talk over a local Unix socket, the calculus changes. We found that disabling keepalives between our SSL terminator and front-line (FL) HTTP processor cut the 99.9th percentile latency by 4x.

The background context matters: Linux socket balancing tends to funnel most connections to a few workers. If you haven't seen it, read up on why one NGINX worker ends up taking all the load. This post is an adaptation of an older internal writeup, so some production details have shifted, but the underlying lesson holds.

The edge architecture

Two components of our edge stack matter here:

  • FL accepts plain HTTP and runs the main request logic, including our WAF.
  • SSL terminates TLS and forwards connections to FL over a local Unix socket.

Both are multiprocess: a main process spawns workers that do the actual work. These days all traffic routes through SSL for simplicity, but the broader point stands regardless of routing details.

The case for keepalives

Reusing connections is generally worth it. New TCP connections are expensive, and servers keep per-connection memory pools that need recycling. We even disable keepalives for abusive clients as a mitigation, forcing them to renegotiate connections and slowing them down.

A quick curl example shows why. The first request to example.com took 18.7ms, of which 12.1ms was connection establishment — and that's without TLS. A second request on the same connection took only 7.3ms. Add TLS 1.3 or QUIC to the mix and the gap widens, plus DNS lookups for low-TTL domains add more pain. Keepalives are enabled by default in most setups for good reason.

The load imbalance problem

Linux's socket handling means most connections land on a small subset of workers. When a busy worker can't accept(), the connection cascades down until an idle worker picks it up. The result is a ladder of load: some workers pegged, others sitting idle.

The relationship is self-reinforcing. Workers with more open sockets generate more load, which keeps them busy, which means they keep more connections. A vicious cycle.

Worse, keepalive connections pin future requests to the same worker. In FL, some request processing is compute-intensive, so a worker can be occupied for a while while others sit idle. That directly hurts latency. Ideally every request lands on an idle worker, even if serving it takes several event loop iterations and blocks downstream.

One long-term fix is offloading compute-heavy work to a thread pool, but that wasn't in place yet. We needed an interim solution.

The counter-intuitive insight

Real network clients can't know which worker is busy, and by the time they reach us tens of milliseconds later, the situation may have changed anyway. But our SSL-to-FL clients aren't far away — they're local, over a Unix socket. Reopening a connection there is nearly free.

Disabling keepalives between SSL and FL means every request arrives on a fresh connection. That gives two advantages:

  • The request is always grabbed by an idle worker.
  • The fully buffered request is immediately available for potential compute-intensive processing.

The first point matters most.

Validation with a synthetic test

To test the hypothesis, we wrote a benchmark that connects to a URL, makes X requests, then closes and reconnects — with short pauses between requests.

Closing after every request produced occasional slow responses on a Lua-served endpoint. That's partly luck of the draw, since the benchmark sees no cooperation from real eyeballs or SSL behavior. But closing after every second request immediately made things worse. Closing after every fifth request nearly doubled the number of slow responses. Counter-intuitive, yes — keepalives were supposed to help latency.

Production results

We disabled keepalives between SSL and FL in one location, forcing SSL to open a separate local connection per request. The effects were dramatic:

  • Cumulative wait time between SSL and FL dropped sharply.
  • The 99.9th percentile wait time improved by roughly 4x.
  • Average edge processing time — including WAF — in the test location fell below the global average.

We cut needless waiting from load imbalance without increasing CPU usage. That translates directly to better user experience and lower resource costs.

Trade-offs and the full fix

There was a downside: CPU imbalance between individual cores increased. Total CPU usage stayed the same, only its distribution shifted. The known remedy is SO_REUSEPORT, though it doesn't work for Unix sockets. An alternative is the EPOLLROUNDROBIN kernel patch, which avoids some of SO_REUSEPORT's limitations but requires a patched kernel.

Combining disabled keepalives with EPOLLROUNDROBIN gave the best outcome — CPU utilization across FL workers converged nicely. Using either change alone was less beneficial for latency.

The takeaway

Disabling keepalives between same-host services was an unexpected fix, but it worked and we can explain why. Local connection setup is cheap enough that the latency risk of landing on a busy worker far outweighs it.

This isn't a recommendation to disable keepalives broadly. For clients across a network, keepalives remain valuable and should stay on. The lesson is context-specific: when connection establishment is nearly free and load balancing is imperfect, fresh connections can be the better choice.

A practical consequence: we can run machines hotter without latency climbing as steeply as before. Push the CPU cap from 50% to 80% with no latency penalty — the exact numbers are illustrative, but the idea holds. Hotter machines with fewer of them serving the same traffic means a smaller overall footprint.