Memory-efficient rollouts for a 60 GiB NGINX process

Cloudflare’s edge network handles roughly 45 million HTTP requests per second on average (61 million at peak) across more than 285 cities. Under that load, even routine operational tasks can become bottlenecks. One recurring problem was deploying new versions of FL (“Front Line”), the company’s oldest proxy application and one of the last major NGINX-based services at Cloudflare.

FL carries a large share of Cloudflare’s business logic, drawing on Lua and Rust libraries. Each instance consumes between 50 and 60 GiB of RAM. Releases relied on NGINX’s standard binary upgrade procedure, which temporarily runs old and new processes side by side. That double memory footprint, sometimes as much as 120 GiB per server, often exceeded available capacity and delayed releases. Engineers rebuilt the upgrade flow to work worker-by-worker, eliminating the need for extra memory during rollouts.

How NGINX upgrades normally work

An FL instance runs a master process that supervises a pool of worker processes. The worker count is tied to the host’s CPU cores — typically half — so a 128-core machine runs 64 FL workers. The standard NGINX upgrade relies on process signals to swap versions:

  1. Send USR2 to the master to spawn a new master process and a full set of new workers.
  2. Send WINCH to have the old workers shut down gracefully.
  3. Send QUIT to stop the original master process.

During the overlap, both versions share the same listening sockets and accept traffic. Memory usage spikes to roughly double the steady-state footprint. With multiple releases per week across thousands of servers, that approach wastes significant memory and occasionally strands servers in an upgrade state for hours because enough free RAM to start the reload isn’t available.

Worker-level control

The core issue is that NGINX treats an instance as a single unit: the upgrade path starts and stops all workers at once. Cloudflare modified NGINX to handle workers individually by introducing new signals recognized by the master process and the workers.

A key complication: NGINX’s master restarts any worker that exits unexpectedly. During a worker-by-worker rollout, exiting workers are intentional, so Cloudflare added a signal to disable auto-restart behaviour while a single worker is being taken down.

The sequence for a server with 64 workers looks like this:

  1. Start with 64 workers from the old version.
  2. Disable auto-restart.
  3. Shut down one old worker, bringing the count to 63.
  4. Launch a new instance with a single new worker, returning to 64.
  5. Re-enable auto-restart.
  6. Repeat, replacing each old worker with a new one.

Memory use stays flat during the whole procedure because at no point are two full worker pools alive.

Order matters: CPU pinning

FL workers are pinned one-to-one onto CPU cores. The operators deliberately terminate an old worker before spawning a new one. Starting a new worker first would briefly leave two processes on one core, skewing CPU load and affecting request latency.

The rollout walks through the cores in a fixed order. For each core:

  1. Shut down the worker pinned to it.
  2. Start a replacement, which NGINX pins to the now-free core.

This discipline ensures that at no point do two overlapping worker generations share a core.

What changed

By shifting from whole-instance swaps to worker-by-worker migration, Cloudflare removed the release-time memory ceiling for FL. Code can now ship to the edge regardless of how much free RAM a server happens to have, and without the hours-long waits sometimes caused by insufficient memory under the old scheme.