Why Cloudflare built a new proxy
Cloudflare’s edge network handles a massive volume of proxied traffic between Internet clients and origin servers. For years, that workload ran on NGINX, but as the company scaled, the limits of that architecture became increasingly costly to work around. The result is Pingora, a Rust-based HTTP proxy that now serves over 1 trillion requests per day while using roughly one-third of the CPU and memory resources of the previous system.
The decision to replace NGINX wasn’t about performance alone — though that was a major factor. It was also about the structural constraints that made certain optimizations impossible and certain features painful to implement. Here’s what pushed Cloudflare to build its own proxy from the ground up.
The performance ceiling of the worker model
NGINX uses a process-based worker model, where each request is handled by a single worker. At Cloudflare’s scale, that creates three significant problems.
First, it leads to unbalanced CPU load. Because a request is pinned to one worker, connections aren’t distributed evenly across all cores. Some workers sit idle while others are saturated, which directly translates into slower request handling.
Second, because each worker operates independently, a request that performs CPU-heavy work or blocking I/O can slow down other requests on the same worker. Cloudflare has spent considerable engineering effort over the years building workarounds — but no workaround eliminates the underlying issue.
The third problem is the most consequential for proxying: poor connection reuse. NGINX keeps its connection pool per worker. When a request lands on a given worker, it can only reuse connections that were established by that same worker. Scaling out by adding more workers actually makes this worse, because the connections become scattered across more isolated pools. The result is higher time-to-first-byte (TTFB) and more origin connections needing to be maintained — a cost borne by both Cloudflare and its customers.
A process model also makes resource sharing difficult. If the goal is to naturally resolve these problems, a shared, multithreaded approach is the more direct path.
Beyond what NGINX was designed for
NGINX is well-suited to being a web server, load balancer, or simple gateway. But Cloudflare’s products — CDN, Workers fetch, Tunnel, Stream, R2 — require far more complex behavior. The company frequently needed to extend NGINX in ways that its architecture wasn’t designed to accommodate.
A concrete example: when retrying or failing over a request, Cloudflare sometimes needs to send it to a different origin with a different set of request headers. NGINX doesn’t allow that. Workarounds exist, but they add complexity and consume engineering time that could go toward product features.
The implementation languages presented their own trade-offs. NGINX is written in C, which is not memory-safe by design — an inherent risk when working with a large third-party codebase. Lua, used to complement C, is less risky but slower, and lacks static typing, which becomes a real liability in complex business logic. Meanwhile, the NGINX community was not particularly active, and development tended to happen behind closed doors.
The build-versus-borrow decision
For several years, Cloudflare evaluated three paths on a quarterly basis:
- Continue investing in NGINX, potentially forking it entirely. The team had the expertise, but the architecture limitations meant significant rework to fully meet their needs.
- Migrate to another third-party proxy like Envoy. This was viable but risked repeating the same cycle in a few years.
- Build an in-house proxy platform from scratch. The highest upfront cost, but the only option that fully addressed the architectural constraints.
For a long time, the path of least resistance won out. But eventually, the return on investment of owning the entire stack became clear, and the Pingora project was born.
Key design decisions
Pingora’s architecture rests on four foundational choices.
Rust for safety and performance
Rust was selected as the implementation language because it provides C-level performance with memory safety guarantees. This directly addresses one of the major risks of working with NGINX’s C codebase.
A custom HTTP library
Rather than using an off-the-shelf library like hyper, Cloudflare built its own HTTP implementation. The reason comes down to the reality of Internet traffic: Cloudflare routinely encounters HTTP that doesn’t strictly follow RFC specifications, and must support it anyway.
A telling example was HTTP status codes. RFC 9110 defines them as three-digit integers, generally expected to fall between 100 and 599. But many servers use codes from 599 to 999. hyper initially rejected such codes, and while the maintainers ultimately accepted the change, the episode illustrated the tension between strict specification compliance and real-world compatibility. For Cloudflare, only a permissive, customizable HTTP library would do — and building their own was the surest way to get one.
Multithreading with work stealing
In place of NGINX’s process-per-worker model, Pingora uses multithreading with work stealing. This allows resources — especially connection pools — to be shared across all threads, solving the connection reuse problem at the architectural level. The Tokio async runtime was chosen as the scheduling foundation.
A familiar, event-based interface
Pingora exposes a “life of a request” event-based programmable interface, similar to NGINX/OpenResty. For example, a “request filter” phase lets developers modify or reject a request when its headers are received. This design keeps business logic separate from generic proxy logic. It also means engineers who already know NGINX can become productive with Pingora quickly.
What the production numbers show
Pingora now handles nearly every HTTP request that needs to reach an origin server, and the production data shows clear gains. Across all traffic, median TTFB is down 5ms and the 95th percentile is down 80ms. Those savings don't come from executing code faster — the previous service could handle requests in well under a millisecond. They come from the new architecture's ability to share connections across all threads.
Better connection reuse means fewer TCP and TLS handshakes. Across all customers, Pingora opens only a third as many new connections per second as the old service did. For one major customer, the connection reuse ratio went from 87.1% to 99.92%, reducing new connections to their origins by a factor of 160. Put differently, the switch saves customers and users 434 years of handshake time per day.
Features and efficiency gains
Faster feature development
The developer-friendly interface and removal of prior constraints have made it possible to ship new functionality quickly. HTTP/2 upstream support, for instance, was added without major hurdles and made gRPC available to customers shortly after. Adding the same support to NGINX would have demanded significantly more engineering effort. More recently, Pingora enabled Cache Reserve, which uses R2 storage as a caching layer.
Lower resource consumption
At the same traffic load, Pingora consumes about 70% less CPU and 67% less memory than the old service. Several factors contribute. Rust code runs more efficiently than Lua, but architecture also matters. In NGINX/OpenResty, accessing an HTTP header from Lua means reading from the NGINX C struct, allocating a Lua string, copying the data, and eventually garbage-collecting the string. In Pingora, that's a direct string access.
The multithreading model also improves data sharing. NGINX shared memory requires a mutex lock on every access and only supports strings and numbers. In Pingora, most shared items are accessed directly via shared references behind atomic reference counters. Fewer new connections also save CPU, since TLS handshakes are much more expensive than sending data over an established connection.
Safety at scale
Predicting every edge case in a distributed system processing millions of requests per second is impossible. Fuzzing and static analysis only go so far. Rust's memory-safe semantics eliminate undefined behavior and make crashes far less likely.
That assurance shifts engineering focus to how a change interacts with other services or a customer's origin. Since Pingora's inception, it has served a few hundred trillion requests without a single crash caused by service code.
When crashes are so rare, the ones that do occur tend to reveal other problems. Recently, Pingora crashes led to the discovery of a kernel bug. In the past, hardware issues on a few machines also surfaced this way — whereas previously, ruling out rare memory bugs in software would have required significant debugging.
