Event Loops, Thread Pools, and the CPU-Bound Problem

When Cloudflare started building its infrastructure over a decade ago, the dominant HTTP server was Apache httpd. The company chose NGINX instead, largely because of its event loop architecture. That design allows a single process to handle many connections concurrently by processing each one until it needs I/O, then queuing it until the I/O completes. A multiplexing syscall like epoll or kqueue notifies the loop when any queued I/O finishes, keeping context switches and memory usage low.

Event loops work on a critical assumption: every unit of work must finish quickly. There is no preemption. If one request performs a long CPU-intensive operation or calls a blocking library function, it stalls every other in-flight request sharing that loop. For a reverse proxy that mostly waits on clients and origins, this model is a good fit. But Cloudflare's Web Application Firewall (WAF) presents a challenge: the vast majority of inspections take a few milliseconds, yet a small fraction take much longer. When a worker process has hundreds of requests in flight, one slow WAF evaluation can degrade all of them. Given that a typical web page needs around 70 requests, tail latency matters more than averages.

Options for Handling CPU-Bound Work

Several approaches exist to keep slow tasks off the event loop:

  • Increase worker processes: This only mitigates the symptom. More workers create more kernel pressure and increase lock contention on critical sections, so capacity does not scale linearly.
  • Run a separate WAF service: A thread-based service isolates CPU-heavy work, but it requires migrating existing code and adds IPC costs such as serialization, latency, and new failure modes.
  • Offload to a thread pool: A hybrid model where the event loop hands specific tasks to a dedicated set of threads. It keeps most of the codebase intact and is faster and simpler than an external service.

Cloudflare chose the thread pool approach because the WAF code already existed within NGINX and needed to run as quickly as possible.

Reusing NGINX Thread Pools

NGINX has supported thread pools since 2015 to handle synchronous filesystem operations, which are notoriously awkward to do asynchronously on Linux—files must be read in direct mode, bypassing the page cache. Each worker process spawns a group of threads dedicated to these blocking operations. The event loop pushes work onto a queue, and the threads notify it when the result is ready. Cloudflare had already used and improved this mechanism for its caching layer, and libuv—the event loop behind Node.js—uses a similar pattern for filesystem calls.

Repurposing the existing thread pools for WAF processing was straightforward. The threading model shares almost nothing between the main loop and the worker threads: only a struct describing the operation is sent, and a result is returned. There are tradeoffs, notably memory usage—each thread gets its own Lua VM and compiled regular expression cache. But the code was written assuming no data races, so changing that would require significant refactoring.

The Load Balancing Catch

There is a complication. The epoll load balancing issue that Cloudflare and others have documented means that when multiple processes listen on the same socket, some end up busier than others. An idle event loop accepts new connections freely, and a process whose loop is freed up by offloading CPU work will accept even more connections. Those connections may in turn need WAF processing, causing tasks to queue up waiting for the thread pool.

To address this, Cloudflare applied a kernel patch that adds the EPOLLROUNDROBIN flag, which distributes incoming connections more evenly across worker processes. That patch was essential to making the thread pool approach work in production.

Measured Impact

The results show clear improvements. Consider the 99th percentile of time a request blocked the event loop:

With the WAF offloaded to a thread pool, this metric dropped by 30 to 40 percent for WAF-enabled requests. Non-WAF requests saw a slight degradation, attributable to the kernel having more threads to schedule, but the net effect was strongly positive.

Looking at customer-facing metrics, the Time To First Byte (TTFB) for cache hits improved significantly in the 99th percentile for both WAF and non-WAF requests:

The event loop's freed-up capacity also reduced accept latency—the time between a connection request arriving and the server beginning to process it:

Both TTFB and accept latency dropped substantially, even though the slightly higher scheduler interruption caused by additional threads appears in micro-metrics.

Tradeoffs and Takeaways

Event loops remain a sensible choice for most I/O-bound workloads, especially in microservices architectures. But they are vulnerable when a significant amount of CPU time enters the picture. The right mitigation depends on the workload's characteristics. For Cloudflare, the thread pool was the best fit because the WAF tasks, while too long for an event loop, are still very short overall. For heavier work, a dedicated service is often the better architectural choice.

This case also illustrates an important principle: a solution that worsens one narrow metric can still deliver substantial end-to-end gains. The scheduler overhead from additional threads made the event loop appear slightly slower in isolation, yet the overall improvement in request processing and acceptance was dramatic.