Scaling Anomaly Detection from 10K to 500K Requests Per Second
Cloudflare’s Bot Management platform relies on a “defense in depth” strategy, combining machine learning, heuristics, JavaScript challenges, and other systems. One component, Anomaly Detection, takes a different approach: it models what legitimate user traffic looks like for a specific site, then flags traffic that diverges from that baseline. Because bots are goal-oriented—whether for credential stuffing or scraping—their behavior patterns eventually stand out from real users.
This method offers two key advantages: it resists bots that spoof metadata like user agents, and it can catch never-before-seen bot behavior without explicit training. The system currently handles over 500K requests per second across roughly 310M unique visitors, issuing more than 200K CAPTCHAs per minute—not counting traffic already blocked by other layers. It generates site-specific baselines automatically for every enrolled site, with traffic coming from 140+ countries and 2,200+ ASNs.
How the Detection Logic Works
The core algorithm is Histogram-Based Outlier Scoring (HBOS). While less precise than methods like kNN for local outliers, HBOS scores global outliers in linear time, which is essential at this scale. Detection relies on two components: a site-specific baseline and a per-visitor behavior model.
Baseline data is aggregated in ClickHouse, an open-source columnar database. Visitor behavior is tracked in-memory using Redis, where HyperLogLog structures efficiently estimate unique counts of high-cardinality features. These privacy-sensitive records exist only within a sliding time window and only for opted-in sites. Each detection run produces an outlier score—how anomalous a visitor appears against the site’s baseline—which feeds into the final bot score at the edge.
A Monolithic Start
The original architecture was a single replicated service running on Kubernetes. 
This v1 design was simple and made heavy use of existing infrastructure—perfect for a rapid launch. But it had clear limitations:
- One logical point of failure across the monolith
- No way to scale individual functions (CPU, memory) independently
- The Quicksilver integration was never designed for this workload, destabilizing both systems
It’s tempting to dwell on those flaws, but the system served its purpose: it provided real-world data that made subsequent improvements targeted and measurable. Launching fast and iterating meant fixes were driven by observation, not speculation.
Redis Bottlenecks and a Simple Fix
Early performance profiling pointed squarely at Redis. The first deployment struggled at just 10K requests per second, largely due to heavy use of the PFMERGE command for combining HyperLogLogs. Unlike most Redis commands, PFMERGE runs in linear time—proportional to the number of features multiplied by the window size—making it a serious bottleneck as demand grew.
One optimization was lowering the threshold for promoting sparse HyperLogLogs to dense representations, trading memory for faster merges. But the biggest win came from a much simpler idea: a “recency register,” a cache that limited how often detection logic ran for a given site-visitor pair. Since behavioral patterns take time to establish, the added latency was irrelevant. This single change improved throughput by an order of magnitude.
Redis work is a constant balance between memory and compute. Shard memory limits were tuned empirically based on CPU utilization; higher memory means more tracked visitors and more commands per second. Since each Redis shard is single-threaded, those trade-offs were easier to reason through. On a separate note, the team learned that Redis’s recommendation for human-readable keys didn’t hold up here. By 
The Shift to Microservices
By a certain point, optimizing parts of the pipeline was no longer enough. As adoption grew, so did the load on shared external dependencies. ClickHouse queries multiplied because every replica of the monolith recalculated the same baselines autonomously; the Quicksilver piggybacking made edge updates bloated and unreliable. Notably, each horizontal scale of the monolith increased pressure on ClickHouse even though the work was redundant.
The decision to redesign wasn’t new in concept—early designs anticipated eventual microservices—but real-world data finally justified the investment. Moving baseline computation to a dedicated service removed duplication, cutting load for that operation by 10x.
The team also discovered existing, battle-tested inserter code at Cloudflare for Kafka-to-ClickHouse pipelines. Adapting it saved development time and brought the design in line with wider internal conventions.
The role of Kafka became clear only after committing to ClickHouse for outlier score storage. ClickHouse handles large, infrequent batches far better than rapid small writes. Streaming outlier scores through Kafka made batching natural while providing resilience against transient downtime—something RESTful interfaces couldn’t offer without considerable extra work.
The Split Architecture
The resulting system divides responsibilities across independent services:
- A Detector service that lazily fetches and caches baselines, calculates outlier scores, and publishes them to Kafka
- A Baseline service that generates the site-specific behavior models
- A Publisher service that batches detections from Kafka and pushes them to the edge for bot score calculations
Each service operates independently and tolerates some dependency downtime; replicas and memory allocations are sized to each service’s specific needs, since costs vary widely.
What’s Next
The engineering challenges aren’t over. The problem space is the cross-product of every site and every unique visitor—enormous cardinality. Improvements are focused on smarter traffic sampling, compressed baseline windows, and more memory-efficient data structures.
Accuracy also needs to improve for sites with multiple legitimate traffic profiles. A website’s web traffic and its mobile API traffic look very different, yet both warrant baselines. HBOS works well with a single site-wide profile but struggles when multiple clusters exist. The team is exploring local outlier factor (LOF) detection, which builds baselines reflecting “local clusters” of behavior—allowing it to distinguish human browser use from automated API abuse on the same site. LOF demands more careful engineering for generating and storing sophisticated baselines, but it promises sharper protection for a broader range of deployments.
There is also ongoing investment in the operational side: tools that speed up model experimentation, “shadow” models that evaluate new detection logic behind the scenes, and instant “escape hatches” to prevent unexpected customer impact.
