Retraining Bot Detection for a Mobile-First Web

Cloudflare’s Bot Management product launched with its first machine learning model, ML1, trained primarily on requests identified by common bot user agents. The initial approach was purely static, built on aggregated HTTP analytics metadata, which allowed for rapid experimentation in the early days. Heuristics were layered on top for quick filtering, but ML remained the core of the detection stack. Since then, five additional models have been deployed, each trained on metadata derived from broader traffic patterns across the network.

The composition of that traffic has shifted decisively. Mobile now accounts for more than 54% of requests on Cloudflare's network, and that shift breaks a key assumption in the earlier models. Desktop traffic tends to share consistent characteristics regardless of destination. Mobile apps are more heterogeneous — each is built on distinct APIs and schemas, with its own behavioral fingerprint. A model trained primarily on browser traffic is structurally disadvantaged when asked to evaluate native app traffic.

From Static Model to Production Pipeline

Building a model that handles both worlds required formalizing a pipeline that could be iterated on quickly. The process starts with labeled data — clearly bot traffic identified by heuristics or verified crawlers, and clearly human traffic from challenge-solving or authenticated sessions. Feature selection is driven by ANOVA f-values, with care taken to ensure operating systems, browsers, device types, and bot categories are all represented. The team uses Catboost for binary classification, selecting hyperparameters through the library’s grid and random search functions.

Before any model sees production, it goes through offline monitoring. Predictions from a newly trained model are compared against the production model across validation datasets for different browsers, OSes, and devices. When results are weak, the SHAP Explainer is used to dissect false positives and negatives, guiding adjustments to the training data or feature set. Deployment is staged: first in shadow mode on live traffic without affecting customers, then progressively rolled out from free-plan users to Super Bot Fight Mode and beta customers before a stable release.

The Mobile App Labeling Problem

The bottleneck for improving mobile performance was never feature engineering — it was labeling. To train useful models, you need accurate ground truth for "human" traffic. Browser traffic gives you a natural source. Users who solve a challenge or CAPTCHA provide a clean signal of legitimacy. But CAPTCHAs rely on HTML and JavaScript rendered in a web browser. Native mobile apps often can’t execute those technologies, meaning legitimate mobile app traffic rarely appears in the high-confidence human datasets.

That gap produced a predictable outcome: models performed worse on native mobile traffic because it was systematically underrepresented in training data. Fixing that meant finding alternate ways to identify legitimate mobile app requests. Cloudflare dug through open source code for mobile frameworks and operating systems to learn what legitimate traffic should look like, and worked with customers to identify domain-specific patterns unique to their apps. From this research, multiple new datasets were built and folded into the training pipeline.

Validated Gains on Real Traffic

The results from this retraining are available early adopter beta customers, and the improvement on mobile traffic has resolved one of the more stubborn sources of false positives in the system.

  • A food delivery company reported Android false positive rates dropping to 0.0%.
  • A Web3 platform that had seen false positive rates anywhere from 28.7% to 40.7% on edge-case mobile app traffic saw that figure fall roughly to 0.0%.

Those are the standout examples, but the underlying dynamic is straightforward: training on trusted, validated mobile datasets is what stopped the model from flagging good requests as bad. Bot analytics remain available to all Cloudflare customers at no charge, so teams can evaluate what sort of traffic their mobile properties see today—and whether bot management makes sense for them.