ROFL with a LOL: rewriting an NGINX module in Rust

Why Cloudflare Rewrote Its HTML Rewriter in Rust

Cloudflare’s reverse proxy, FL (Front Line), handles a substantial share of the company's application logic — largely written in Lua on top of OpenResty and NGINX. One of its oldest components, cf-html, parses and rewrites HTML as it streams from origin servers to visitors. Features like email obfuscation, which replaces addresses with JavaScript to deter bots from scraping them, depend on this framework.

For years, cf-html was built on C and a streaming HTML parser called Lazy HTML, with logic generated by Ragel state machines. The drawbacks became impossible to ignore. C code is prone to memory corruption, and in 2017 a flaw in this area led to Cloudbleed, an incident where FL read arbitrary memory and appended it to response bodies, potentially exposing data from other requests. The aftermath shaped Cloudflare’s approach to memory safety, and new features for cf-html were largely shelved as a result.

By 2022, teams were requesting a safe way to inspect and rewrite response body data. Meanwhile, a parallel effort produced lol-html (Low Output Latency HTML), a Rust-based HTML parser already used in production by Cloudflare Workers. Faster than Lazy HTML and memory-safe by construction, it became the obvious foundation for a replacement system: ROFL (Response Overseer for FL), a new NGINX module written entirely in Rust. It is now running in production on millions of responses per second.

Building an NGINX Module in Rust

NGINX provides little documentation for writing modules outside C. The ROFL team leaned on code from the nginx-rs project early on, especially around buffer and memory pool handling, and used Rust's Bindgen library to generate FFI bindings from NGINX header files. After configuring a copy of NGINX, a build.rs file tells Bindgen which directories contain headers and which NGINX symbols to generate bindings for. A wrapper.h file patches a few symbols that Bindgen struggles with.

A generated bindings.rs file then provides Rust definitions for all NGINX symbols, which serve both as a binding layer and as a reference for the internal structure of NGINX objects. This reduces the chance of errors in the interface code versus hand-written FFI.

Dynamic loading is the critical next step. Since NGINX 1.9.11, modules can be compiled separately and loaded via the load_module directive in nginx.conf. To load a Rust library at startup, the module must export the symbols NGINX expects via dlopen. Module ordering also matters: dynamic modules run first on a response, so ROFL needs to specify its position relative to other modules, such as running after gzip decompression, to avoid trying to parse compressed output.

Passing State from Request to Response

NGINX is hostile to external service calls during the response phase. To determine which features to apply to a response, that decision must be made during the request phase and stored. NGINX provides the ctx member on the ngx_http_request_s struct for exactly this purpose — a place to store arbitrary data for the lifecycle of a request. A helper function retrieves the ctx using the module's ctx_index:

pub fn get_ctx(request: &ngx_http_request_t) -> Option<&mut Ctx> {
    unsafe {
        match *request.ctx.add(ngx_http_rofl_module.ctx_index) {
            p if p.is_null() => None,
            p => Some(&mut *(p as *mut Ctx)),
        }
    }
}

Once initialized, this ctx can hold feature flags or other settings set during the request phase. When work happens on the response stream, the module reads the ctx to know which features to activate, avoiding any database calls that would slow things down.

NGINX memory pools further simplify this work. Memory allocated to a pool is automatically freed when a request ends, so ROFL allocates from NGINX pools and registers a cleanup callback rather than managing memory lifetimes manually:

pub struct Pool<'a>(&'a mut ngx_pool_t);

impl<'a> Pool<'a> {    
    /// Register a cleanup handler that will get called at the end of the request.
    fn add_cleanup<T>(&mut self, value: *mut T) -> Result<(), ()> {
        unsafe {
            let cln = ngx_pool_cleanup_add(self.0, 0);
            if cln.is_null() {
                return Err(());
            }
            (*cln).handler = Some(cleanup_handler::<T>);
            (*cln).data = value as *mut c_void;
            Ok(())
        }
    }

    /// Allocate memory for a given value.
    pub fn alloc<T>(&mut self, value: T) -> Option<&'a mut T> {
        unsafe {
            let p = ngx_palloc(self.0, mem::size_of::<T>()) as *mut _ as *mut T;
            ptr::write(p, value);
            if let Err(_) = self.add_cleanup(p) {
                ptr::drop_in_place(p);
                return None;
            };
            Some(&mut *p)
        }
    }
}

unsafe extern "C" fn cleanup_handler<T>(data: *mut c_void) {
    ptr::drop_in_place(data as *mut T);
}

The tradeoff is an interface that requires a fair amount of unsafe code, since Rust must manipulate C constructs across an FFI boundary. The team is planning to reduce that surface area over time.

Lessons from the Trenches

Two problems in particular proved difficult to diagnose. The first involved syncing Rust views with NGINX data structures. NGINX chunked response bodies are stored in linked lists. It is tempting to build a Rust view over those lists for convenience, but any mutation must update both the Rust view and NGINX’s own pointers. An early version of ROFL failed to update which buffer Rust's free_chain.head pointed to, causing an infinite loop that locked up NGINX worker processes. The bug only manifested with larger response bodies, making it nearly impossible to reproduce on a development machine. By the time a coredump could be captured, the process memory had grown too large to write to disk. The code never reached production, but the lesson stuck: Rust's compiler cannot help when shared state comes from an external FFI environment.

The second challenge was backpressure. When ROFL injects a large JavaScript chunk — say, for email obfuscation — the output can grow beyond what NGINX's downstream modules can absorb. If the EAGAIN error from the next module is unhandled, data can be dropped and HTTP response bodies truncated. ROFL addresses this with a dedicated buffer chain called saved_in that temporarily queues output to keep downstream modules from being overwhelmed. This kind of issue is not covered by NGINX's development guide, which sticks to trivial examples.

#[derive(Debug)]
pub struct Chains {
    /// This saves buffers from the `in` chain that were not processed for any reason (most likely
    /// backpressure for the next nginx module).
    saved_in: RefCell<Chain>,
    pub free: RefCell<Chain>,
    pub busy: RefCell<Chain>,
    pub out: RefCell<Chain>,
    [...]
}

A Future Beyond NGINX

Cloudflare has been steadily moving components off OpenResty to Workers and Rust-based proxies. FL, with its dense web of application services logic, is at the hard end of that migration. But the work on ROFL is forward-looking. It was designed to run outside NGINX from day one, making it straightforward to port to a Rust-based proxy or Workers platform later. cf-html and its feature set are a key dependency; having a NGINX-independent version reduces friction for the broader platform migration.

Memory-safe languages are often discussed in terms of bug prevention. Cloudflare frames the value differently: safety enables engineering that would otherwise be too risky. A filter language for firewall rules, executing arbitrary user JavaScript on the platform, or rewriting streaming HTML in place all carry strict boundaries that make them safe to offer. ROFL retired one of the scariest components in Cloudflare's codebase, and the approach it represents is the direction the platform is heading.