Every request, every microsecond: scalable machine learning at Cloudflare

Scaling Machine Learning Inference at Cloudflare

Cloudflare’s edge network processes an average of 46 million HTTP requests per second, peaking above 63 million. Machine learning detection is responsible for classifying the largest share of this traffic — it produces the final Bot Score decision for over 72% of all HTTP requests. In addition, we run multiple models in shadow mode on every request, and predictions are driven by CatBoost, our inference engine of choice.

Latency, however, is not solely a function of model inference. Equally important are the steps that precede it: machine learning feature extraction and preparation. Optimizing these stages has been central to our efforts to scale the number of models and features without degrading request performance.

The Shift to Inter-Request Features

We initially relied on single-request features extracted from individual HTTP request attributes like header presence or specific values. These proved easy to spoof, so we evolved to inter-request features, which aggregate signals across multiple dimensions over a sliding time window. For instance, we may count the number of unique user agents tied to a particular dimension of a request.

Feature extraction for inter-request data was originally handled by Gagarin, a Go-based feature serving platform we built. The flow worked as follows:

1. Extract dimension keys from request attributes as a request arrives at Cloudflare.
2. Look up the correlated machine learning features in a multi-layered cache.
3. If missing, issue a memcached "get" request to fetch the features from Gagarin.
4. Feed the features into CatBoost models to produce detections, surfaced via Firewall and Workers fields.
5. Log results through our ClickHouse logging pipeline for further experiments by data scientists.

Initial performance for Gagarin was acceptable: a median latency of roughly 200 microseconds to serve all features for a given key. As we added dimensions and the traffic profile changed, cache hit ratios dropped. Median latency climbed to 500 microseconds, and at peak traffic p99 latency approached 10 milliseconds. Extensive low-level tuning and benchmarking of Gagarin could not overcome the inherent costs of IPC over Unix Domain Socket (UDS).

Why the Previous Design Broke Down

The limitations of the old architecture fell into several categories:

• High tail latency: Peak-time requests saw increased delays from CPU contention on the Unix socket and Lua garbage collector pressure.
• Suboptimal resource utilization: CPU and RAM usage was not optimal, reducing headroom for other edge processes.
• Feature availability issues: Memcached timeouts lowered feature availability, potentially increasing false positives and false negatives for some requests.
• Hard scalability limits: Adding more features and dimensions pushed the infrastructure toward its capacity ceiling.

Evaluating Alternatives

We considered a range of design directions before settling on a new approach.

• Further optimizing Gagarin: Deeper optimization was bounded by UDS synchronization overhead, multiple data copies, serialization/deserialization cost, Go’s garbage collector, and hashmap lookup performance.
• Quicksilver: Reusing Quicksilver was ruled out — the volume and update frequency of machine learning features would stress capacity and harm other use cases. It also runs over a Unix socket with memcached protocol, inheriting identical limitations.
• Increasing cache size: Enlarging the multi-layered cache to hold tens of millions of dimension keys was impractical due to memory duplication across worker threads.
• Sharding the Unix socket: Sharding eased contention but only partially fixed the problem while adding substantial system complexity.
• RPC-based communication: An RPC layer would still need a transport (TCP, UDP, or UDS), so it offered no meaningful gain over the already minimal memcached-over-UDS approach.

That left us to investigate IPC mechanisms more broadly.

Comparing IPC Primitives

Working from first principles, we asked which low-level data transfer method between processes is most efficient. Using ipc-bench, we benchmarked latency for transferring one million 1,024-byte ping-pong messages between two processes in our test environment.

Unix sockets handle synchronization for you, but they were not the fastest option. Shared memory and memory-mapped files outperformed other mechanisms by a wide margin and were closely matched. Shared memory required allocation on a tmpfs volume in /dev/shm; memory-mapped files could reside on any volume, including tmpfs or disk.

Choosing Memory-Mapped Files

We chose memory-mapped files as our IPC mechanism for serving machine learning features. This choice cut latency, lowered CPU contention, and minimized data copies. But unlike a Unix socket, a memory-mapped file is just a file — no synchronization mechanism is built in. Before we could proceed, several critical design questions had to be answered:

1. How do we efficiently fetch hundreds of float features for given dimension keys from a file?
2. How do we handle safe, concurrent, and frequent updates for tens of millions of keys?
3. How do we prevent the CPU contention seen with Unix sockets?
4. How do we keep adding dimensions and features without hitting a structural limit?

Addressing these required several additional ingredients to be added to the overall recipe.

Making concurrent reads wait-free

Building a memory-mapped feature store requires solving the concurrency problem: how do you let many threads read from a structure while a writer updates it, without locks serializing access? Three approaches exist. With-lock synchronization — mutexes or spinlocks — guarantees mutual exclusion but suffers from contention and blocking. Lock-free synchronization uses atomic operations to ensure at least one thread progresses. Wait-free synchronization goes further: every thread is guaranteed to complete its operation in a bounded number of steps, regardless of what other threads do.

Cloudflare's design borrows from the Linux kernel's Read-Copy-Update (RCU) pattern and the Left-Right concurrency control technique. The implementation keeps two copies of the data in separate memory-mapped files. A single writer manages updates while multiple readers access the data concurrently. A third memory-mapped file, called "state", coordinates access between the two copies. It holds an atomic 64-bit integer representing an InstanceVersion, plus two atomic 32-bit counters tracking active readers for each data copy. The InstanceVersion packs three pieces of information: the currently active data file index (1 bit), the data size (39 bits, supporting data up to 549 GB), and a 24-bit data checksum.

	Disjoint Access Parallelism	Starvation Freedom	Finite Execution Time
With lock
Lock-free
Wait-free

Zero-copy deserialization with rkyv

Deserialization latency is another bottleneck when fetching machine learning features. Zero-copy deserialization sidesteps it by referencing bytes directly in the serialized form, eliminating both the copy and the parsing work. The team chose rkyv, a Rust framework that implements total zero-copy deserialization. rkyv structures its encoded representation to match the in-memory layout of the source type, meaning no work is performed at read time.

A standout capability drove the selection: rkyv can access HashMap data structures in a zero-copy fashion, something no other Rust serialization library offers. The project also benefits from an active Discord community that provides guidance and accepts feature requests.

Feature comparison: rkyv vs FlatBuffers and Cap'n Proto

The mmap-sync crate

These three concepts — memory-mapped files, wait-free synchronization, and zero-copy deserialization — are packaged into an open-source Rust crate called mmap-sync. The core structure is Synchronizer, which reads and writes any data expressible as a Rust struct. Users derive a single Rust trait on their struct definition, then interact with two methods: "write" and "read".

impl Synchronizer {
    /// Write a given `entity` into the next available memory mapped file.
    pub fn write<T>(&mut self, entity: &T, grace_duration: Duration) -> Result<(usize, bool), SynchronizerError> {
        …
    }

    /// Reads and returns `entity` struct from mapped memory wrapped in `ReadResult`
    pub fn read<T>(&mut self) -> Result<ReadResult<T>, SynchronizerError> {
        …
    }
}

/// FeaturesMetadata stores features along with their metadata
#[derive(Archive, Deserialize, Serialize, Debug, PartialEq)]
#[archive_attr(derive(CheckBytes))]
pub struct FeaturesMetadata {
    /// Features version
    pub version: u32,
    /// Features creation Unix timestamp
    pub created_at: u32,
    /// Features represented by vector of hash maps
    pub features: Vec<HashMap<u64, Vec<f32>>>,
}

A read operation performs zero-copy deserialization and returns a "guarded" Result holding a reference to the Rust struct via the RAII pattern. The operation increments the active reader counter; when the Result goes out of scope, the counter is decremented. The synchronization mechanism is wait-free, guaranteeing an upper bound on steps per operation. Because data lives in shared mapped memory, the Synchronizer can write and read concurrently.

BLISS: a system redesign

The previous architecture fetched machine learning features from a Lua module making memcached requests over a Unix socket to a Go service called Gagarin. The redesign pivots on mmap-sync and introduces two components forming the Bots Liquidation Intelligent Security System — BLISS.

Bliss service

The bliss service is a Rust-based, multi-threaded sidecar daemon built for batch processing large data volumes and heavy I/O. It fetches, parses, and stores machine learning features and dimensions. The Tokio event-driven platform provides the non-blocking I/O foundation.

Bliss library

The bliss library is a single-threaded dynamic library integrated into each worker thread via the Foreign Function Interface (FFI) through a Lua module. It avoids heavy I/O entirely, serving machine learning features and generating detections with minimal resource usage and ultra-low latency. Several performance techniques reinforce the design:

• Allocations-free operation: the library re-uses pre-allocated data structures, performing no heap allocations. Integration tests use the dhat heap profiler to enforce the zero-allocation policy.
• SIMD optimizations: vectorized CPU instructions — AVX2 and SSE4 — accelerate hex-decoding of certain request attributes by tenfold via the faster-hex crate.
• Compiler tuning: both the service and library compile with performance-focused flags.

[profile.release]
codegen-units = 1
debug = true
lto = "fat"
opt-level = 3

• Benchmarking & profiling: Criterion benchmarks every major feature. The Go pprof profiler runs on Criterion benchmarks to produce flame graphs.

cargo bench -p integration -- --verbose --profile-time 100

go tool pprof -http=: ./target/criterion/process_benchmark/process/profile/profile.pb

Rollout results

The migration preserved full backward compatibility — no customer-reported false positives or negatives emerged during the transition. The latency gains were substantial. Cloudflare's overall HTTP request processing latency improved by an average of 12.5%. Within the Bot Management module specifically, latency improved by 55.93%.

Bot Management module latency, in microseconds.

Machine learning feature fetch latency improved by several orders of magnitude:

The aggregate impact is striking: at Cloudflare's average rate of 46 million requests per second, a 523 microsecond saving per request amounts to over 24,000 days — 65 years — of processing time saved every single day.

Additional benefits accompanied the latency reduction:

• Enhanced feature availability: eliminating Unix socket timeouts raised machine learning feature availability to 100%, reducing false positives and negatives in detections.
• Improved resource utilization: the overhaul freed thousands of CPU cores and hundreds of gigabytes of RAM across the server fleet.
• Code cleanup: thousands of lines of less performant, less memory-safe Lua and Go code were removed, reducing technical debt.
• Upscaled machine learning capabilities: inference now handles hundreds of machine learning features and dozens of dimensions and models.

Building on the platform, Cloudflare is deploying a new machine learning model built on BLISS with select customers. Bot Management subscribers interested in testing it can contact their account team. The mmap-sync crate is available as open source for the wider community.