What counts as a bot

A bot is an autonomous program on a network that can interact with computer systems or users, imitating or replacing human behavior and performing repetitive tasks much faster than a person could. In practice, bots split into two camps: good bots that serve the businesses they touch, such as search engine crawlers like Googlebot and Bingbot or social platform bots like Facebook Bot, and bad bots built for malicious ends — credential stuffing, third-party scraping, spam and sneakerbots.

Bot Management means blocking undesired or malicious bot traffic while still letting useful bots through: detecting bot activity, telling desirable from undesirable behavior, and identifying where the undesirable activity originates. It sits alongside the WAF, the rule-based security system that monitors and controls network traffic.

Requirements that shaped the platform

Cloudflare had blocked malicious bots since its earliest days, but over time it became clear a dedicated platform was needed to unite the various bot-fighting techniques and simplify the customer experience. The product goals were:

  • Complete, not complex — protection for websites, mobile applications and APIs toggled with one click.
  • Trustworthy — a certainty indicator telling customers whether a visitor is who they claim to be.
  • Flexible — customers choose which traffic subset mitigations apply to, e.g. only login URLs, pricing pages, or sitewide.
  • Accurate — almost no human visitors should ever be misclassified as bots.
  • Recoverable — when a prediction is wrong, humans and good bots must still get through.
Cloudflare Bot Management: machine learning and more Embedded Image - kzsrPf

Engineering added its own must-haves: scalability to score every request even above 10 million requests per second; low latency that adds no more than 100 microseconds to request processing and needs no extra hardware; configurability per domain, data center or server; modifiability so new detections, mitigation actions, analytics and logs can be added; security so no customer's sensitive data trains models protecting another customer; and explainability and debuggability so predictions can be explained and tuned intuitively. In 2018 a small team set out to build this next generation of Bot Management.

The score as the product's core

Cloudflare covers 26M+ Internet properties, processes an average of 11M requests per second (peaking above 14M), and examines more than 250 request attributes across protocol levels. The design question was how to turn that data into protection that is simple, reliable and explainable, given that some bots only reveal themselves across many dimensions of request attributes over time, while others give themselves away through a single attribute.

The answer was the Score: a single number from 0 to 100 expressing the likelihood that a request came from a human (high) rather than an automated program (low). Exposing it only in logs would have forced customers into manual, error-prone, non-real-time workflows — finding low-scoring IPs and blocking them after the fact. Instead, the score became a field in Firewall Rules, which Cloudflare's Firewall team had just announced, offering flexible request control inspired by the Wireshark language. Thus cf.bot_management.score was born.

Putting the score at the center satisfied several requirements at once: it is simple, flexible and configurable, it gives per-request telemetry on bots, customers can set thresholds in firewall rules according to their tolerance for false positives and negatives, and detection capabilities can grow underneath without requiring configuration changes.

Architecture and the edge module

The platform is a set of microservices, reusing existing pipelines, databases and components where possible and building many services from scratch. The stack spans Kafka, ClickHouse, Postgres, Redis and Ceph for data; Go, Rust, Python, Java and Bash for code; Salt, Quicksilver and Cap'n Proto for configuration and schemas; and Docker, Kubernetes, Helm and Mesos/Marathon for containerization. Each service is built with resilience, performance, observability and security in mind.

Cloudflare Bot Management: machine learning and more Embedded Image - ML6asS

What runs on every request

Detection happens in real time during request processing, in a Bot Management module running on every machine at Cloudflare's edge locations. Request attributes are extracted and transformed, fed to the detection mechanisms, and the module emits:

  • Firewall fields — cf.bot_management.score (the 0–100 integer), cf.bot_management.verified_bot (boolean, whether the request comes from a Cloudflare allowlisted bot) and cf.bot_management.static_resource (boolean, whether the request matches file extensions for many types of static resources).
  • Cookies — most notably cf_bm, which helps manage incoming traffic matching bot-related criteria.
  • JS challenges — invisible challenges injected for some detections and customers, yielding additional signals.
  • Detection logs — details about each applied detection, used features and flags, piped to ClickHouse for analytics, customer logs, debugging and model improvement.

Mitigation itself is the Firewall's job: every request is inspected against the rule engine, and requests can be blocked, challenged or logged based on the score and configured threshold.

Actions, tooling and Workers

Bot mitigation is expressed through firewall rule actions:

  • Log — records matching requests in the Cloudflare Logs provided to customers.
  • Bypass — dynamically disables Cloudflare security features for a request.
  • Allow — exempts matching requests from challenge and block actions triggered by other Firewall Rules content.
  • Challenge (Captcha) — verifies the visitor is human, not automated program.
  • JS Challenge — keeps bots and spam from the requested resource while letting browsers satisfy the challenge automatically.
  • Block — denies access to the site.

The dashboard supports rule building through clicks, with Terraform integration. Firewall Analytics, powered by ClickHouse and a GraphQL API, supports threat investigation, and detailed bot-related logs are available through the Logpull API and LogPush. Customers needing more flexibility — injecting or altering HTML content, serving incorrect data to bots, stalling requests — can use Cloudflare Workers; a small snippet can, for example, pass the score back to the origin for advanced real-time analysis or mitigation.

Cloudflare Bot Management: machine learning and more Embedded Image - Yqtziq
addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})
 
async function handleRequest(request) {
  request = new Request(request);
 
  request.headers.set("Cf-Bot-Score", request.cf.bot_management.score)
 
  return fetch(request);
}

Five detection mechanisms

Five complementary mechanisms each produce their own score, combined into the single score sent to the Firewall. Most run on every request; some are enabled per customer. Scoring every request for every customer brings its own benefits, from easy onboarding (historical bot activity is visible before active mode is enabled) to a feedback loop for continuous improvement, guaranteed scaling, and global visibility into threat shifts. Globally, more than a third of traffic visible to Cloudflare comes from bad bots; among Bot Management customers the share is higher still, at ~43%.

Cloudflare Bot Management: machine learning and more Embedded Image - PZ7Hne
Cloudflare Bot Management: machine learning and more Embedded Image - dCpemL
Cloudflare Bot Management: machine learning and more Embedded Image - 2BITb9
Cloudflare Bot Management: machine learning and more Embedded Image - f6i9yU
Cloudflare Bot Management: machine learning and more Embedded Image - oUzqMk

Machine learning

Machine learning made the majority of score decisions and was the first detection mechanism, producing scores and onboarding customers in 2018. It rests on CatBoost, an open source gradient boosting library for decision trees, chosen for categorical feature support (even at very high cardinality), accuracy through a gradient-boosting scheme that reduces overfitting, inference under 50 microseconds per model, and a C and Rust API that fits the LuaJIT edge code via FFI. Multiple CatBoost models run on the edge in shadow mode on every request on every machine; one runs in active mode and influences the final Firewall score. Results and features are logged to ClickHouse, drawing on categorical and numerical features built from request attributes and from inter-request features computed by the Gagarin platform. New models deploy in seconds via Quicksilver, which can also select the active model version per customer.

Heuristics engine

Added in 2019 as the second mechanism and run on every request, the heuristics engine recognizes that some bots are given away by a single request attribute. Hundreds of rules across multiple heuristic types match on hard-to-spoof attributes; any match assigns the lowest possible score, 1. The engine's hundreds of heuristics apply in just under 20 microseconds, new heuristics deploy in seconds via Quicksilver, and a conservative design gives it the lowest false positive rate of all mechanisms. It classifies ~15% of global traffic and ~30% of Bot Management customers' traffic as bots, and because its matches are high-certainty they also serve as labels for training the ML models.

Behavioral analysis

Developed in 2018 and integrated in 2019, behavioral analysis is unsupervised: rather than relying on a teacher for labels, it learns normal visitor behavior for each customer over an extended period. It is automatically enabled for all Bot Management customers, detects bots never seen before by flagging anomalies against normal behavior, and is harder for bots to evade because anomalous behavior is often a direct byproduct of the bot's goal. More detail is promised for a later post.

Verified bots

Most good bot requests already score low, but the cf.bot_management.verified_bot field, introduced in 2019, prevents accidental blocks of useful bots and lets customers choose whether to admit all good bots or restrict them from parts of the site. The platform takes a validator-based approach, clustering sets of IPs to confirm bot identity through: a reverse DNS validator matching IP against alleged hostname; an ASN block validator performing a similar check on the ASN block; a downloader validator collecting good bot IPs from text files or HTML pages on the bot owner's site; and a machine learning validator clustering IPs that no other means can validate. A Bots Directory, a database with a UI, stores and manages bots passing through the network. Together these identify hundreds of unique good bot identities across companies and categories.

Cloudflare Bot Management: machine learning and more Embedded Image - z77XmW

JS fingerprinting

This mechanism draws on client-side signals: browser graphics output such as canvas depends on hardware (GPU) and software (drivers, operating system rendering), and the resulting unique output allows precise differentiation of browser and device types without sacrificing visitor privacy — it is not a supercookie and cannot track or identify individuals, only confirm that the user agent matches other telemetry gathered through the canvas API. It works as a challenge-response system, with a challenge injected at the edge, rendered in the background from provided graphics instructions, and returned to Cloudflare for validation. The system is in private beta, with promising early results; wider availability and an engineering post are still to come.

Closing view

Cloudflare collects data from trillions of requests flowing through its network every week; from that data it identifies likely bot activity through machine learning, heuristics, behavioral analysis and the other mechanisms described here, with Bot Management integrating seamlessly with products such as WAF and Workers. The platform depended on work across teams beyond Bots, most notably Firewall, Data, Solutions Engineering, Performance and SRE. Further posts in the Bots series will cover the internals of each detection mechanism.