How Cloudflare runs machine learning inference in microseconds

Cutting Latency From the Bot Management Hot Path

Cloudflare’s edge servers enforce a broad set of security checks on every request. Each check adds compute time to the request-processing path, and with a global network that handles a huge and diverse volume of traffic, that time must be carefully managed. Latency is a core metric for CDNs, and in addition to optimizing network distance, Cloudflare also focuses on processing latency — the time spent on a request before a response is served or forwarded.

The bot management module is one of the heavier security suites running in this hot path. It evaluates every request for bot-related signals, scoring traffic with several machine learning models. These model executions account for roughly 15% of the latency introduced by bot detection.

Cloudflare recently rewrote the bot management module, moving from Lua to Rust. This article details the performance work done on the machine learning components of that rewrite. By switching away from a garbage collected language, eliminating memory allocations, and tightening parser code, the team cut the module’s P50 latency by 79μs — a 20% reduction.

Designing for Zero Allocations

Memory allocation is a hidden cost in many high-level languages. Garbage collected runtimes track memory usage, pause execution for collection cycles, and rely on allocators that scan for contiguous free regions and occasionally request new memory pages from the kernel. When latency is the priority, low-level control over memory is often necessary.

There are several practical strategies for avoiding allocations in Rust:

Use Stack Buffers

Fixed-size buffers can live on the stack, which simply reserves space in the current stack frame. No heap allocator logic is involved. Alternatively, a buffer can be allocated once outside the hot path and reused for the lifetime of the process.

The performance difference is measurable. Two implementations of a case-insensitive string equality check — one allocating a fresh buffer per call, the other reusing an existing buffer — benchmark at ~40ns and ~25ns per iteration, respectively. The allocation-free version is roughly 38% faster.

Pick Allocation-Free Algorithms

Where possible, algorithms should process data in place and hold state on the stack. A stack-only, no-copy rewrite of the string comparison runs at ~13ns per iteration, close to the ~11ns of Rust’s standard library eq_ignore_ascii_case, which likewise avoids buffering through iterators.

Test for Allocations

To prevent regressions, allocation behavior can be enforced with the dhat crate. By setting dhat as the global allocator, code can count allocations and allocated bytes along a given path. One limitation: dhat only observes allocations from Rust code. C or other FFI calls can allocate without being detected.

Zero-Allocation Decision Trees

The bot management module runs models built with CatBoost, an open source machine learning library implemented in C++ with bindings for C, Rust, and other languages. The previous Lua implementation called the C API through FFI to execute models.

By removing allocations and reusing buffers, Cloudflare cut execution time for a sample CatBoost model by 10%, with production models seeing improvements up to 15%.

Specialize for Single-Document Evaluation

CatBoost’s API includes routines that handle multiple documents at once, but those are a poor fit here. Requests arrive sequentially, so batching would only add delay. The multi-document path also allocates vectors and copies each document’s features into them.

For a single document, the model only needs a reference to contiguous memory containing the feature values. The same applies to the Rust bindings, which allocate an outer vector of pointers for multi-document input. In the single-document case, that outer vector is unnecessary — the inner pointer can be passed directly.

Reuse Buffers and Pre-Hash Features

Additionally, the previous Rust binding API took ownership of Vecs, which forces the function to free memory on exit and rules out buffer reuse. Categorical features were passed as owned Strings, requiring a temporary heap allocation and a byte copy for each request, rather than borrowing byte slices from the original request.

Rewriting the function signatures to accept &[f32] and &[&str] avoids those copies. A further optimization is to compute categorical feature hashes once and reuse them across the multiple models that share those features per request, instead of re-hashing for each model.

Results

These changes reduced the bot management module’s P50 latency from 388μs to 309μs (a 20% drop) and P99 latency from 940μs to 813μs (a 14% drop), all while keeping request handling on the critical path allocation-free. In many cases, the optimized code is also shorter, which makes it easier to read and maintain. Further details on the broader Lua-to-Rust port of bot management are available in a separate engineering post on Cloudflare’s blog.