Per-Customer Bot Defenses in the Age of AI Scraping
Bot management has traditionally been a game of global pattern matching: spot the suspicious request, block the known bad actor, and repeat. But the rise of AI-driven scraping has changed the rules. Attackers no longer rely on clumsy scripts with obvious flaws. Instead, they deploy headless browsers, rotate residential proxies, and generate human-like user agents to blend into normal traffic. A request from one of these bots might look perfectly reasonable next to the trillions of requests Cloudflare sees across its network, yet stand out clearly against the behavioral patterns of legitimate users on a single website.
To address this, Cloudflare is announcing a new approach: behavioral anomaly detection that is unique to each bot management customer. Instead of relying solely on a first-request judgment call, the platform now tracks the behavior of bots that play a longer game, continuously executing unwanted actions on customer sites. The result is a per-customer detection suite, with the capacity to fuel hundreds of thousands of unique configurations. The initial focus, based on direct feedback from site owners, is protecting websites from the surge of sophisticated, AI-driven web scraping.
Why Traditional Bot Detection Falls Short
The earlier era of bot defense was simpler. Attackers used static, predictable signals—missing User-Agent headers, malformed method names, or traffic from non-standard ports—that were easy to spot. As websites became more dynamic, attackers upgraded to headless browsers and automation frameworks capable of rendering pages and mimicking human interaction with far greater fidelity.
Generative AI has accelerated this evolution. Web scraping is no longer limited to price intelligence or content aggregation; it is now driven by the appetite of Large Language Models (LLMs) for training data. Cloudflare's network data shows the scale of this shift: in mid-2025, crawling for AI model training accounted for nearly 80% of all AI bot activity on the network, a major jump from the previous year.
Modern scraping tools are themselves AI-powered. They use LLMs for semantic understanding of page content, computer vision to solve visual challenges, and reinforcement learning to navigate unfamiliar websites. These bots are designed to blend in, rotating IP addresses through residential proxies and mimicking plausible browsing patterns. This is where the global, one-size-fits-all approach exposes a critical vulnerability: a bot that looks normal across the entire network can still be detected as anomalous when compared to the established habits of real users on a specific application.
Layering Global Intelligence with Local Behavior
To counter well-known bots and distributed botnets, Cloudflare continues to leverage its global vantage point. Since June, bot detection analysts have written 50 heuristics to catch bots using signals like HTTP/2 fingerprints and Client Hello extensions. By observing traffic across millions of sites, the network builds a baseline of legitimate fingerprints for common browsers and benign devices. When a new, unique fingerprint suddenly appears across many different sites, it signals a distributed botnet or a new automation tool. Blocking the bot's signature can neutralize an entire campaign, regardless of the thousands of IP addresses it uses.
Cloudflare has also introduced detection improvements to tackle residential proxy networks, which attackers use to make bots appear as thousands of distinct real visitors. The approach combines network-wide data with client-side fingerprints obtained from the millions of challenge solves that occur daily. Challenges have always been a mitigation action that protects applications without compromising real-user experience, but they now feed threat detection teams a constant stream of client-side information. This allows the platform to pattern match and identify IP addresses used by residential proxy networks.
This detection improvement is already live across the Cloudflare network, automatically catching more malicious traffic for customers using Super Bot Fight Mode and Enterprise Bot Management. In the seven days since the feature was authored, Cloudflare observed 11 billion requests from millions of unique IP addresses identified as connected to residential or commercial proxy networks. This complements the existing residential proxy detection features in the ML, which already catch tens of millions of requests every hour.
The new per-customer behavioral anomaly detection adds a second layer to this global fingerprinting. While global intelligence is powerful against widespread attacks, the tailored approach is designed for the scraper that behaves acceptably across the network but abnormally on a specific site. Together, the two layers aim to provide defenses from every angle—from the macro view of the network to the micro view of a single application.
Why one-size-fits-all bot defense is failing
The rise of AI-powered bots has exposed a fundamental weakness in traditional bot management: static, global rules cannot distinguish between a legitimate user and a sophisticated scraper when each individual request looks benign. A script that methodically pulls every product listing from an e-commerce site, or every profile from a social network, only reveals its intent when viewed in the context of that specific application's normal traffic patterns.
Cloudflare's response is a new foundational platform built to deploy custom machine learning models for every bot management customer. The core principle is simple: because each website has different traffic, the definition of anomalous behavior must be unique to each zone. Data from one customer's traffic will not be used to train another customer's models. This is a platform capability rather than a single feature, designed to address today's scraping problems while laying groundwork for future detections as bot tactics evolve.
The three-step detection process
The new per-customer anomaly detection system follows a three-step process to identify malicious behavior by first understanding what legitimate traffic looks like for each individual website and API.
Step 1: Building a living baseline
For each customer zone, the system ingests traffic data to construct a continuously updated baseline of normal activity. This is not a static snapshot; it accounts for seasonality, recognizes legitimate traffic spikes from marketing campaigns, and maps typical user pathways through a site. This evolves the Anomaly Detection already present in the Enterprise Bot Management suite, but applies it at a far more granular, per-customer level.
Step 2: Spotting contextual anomalies
With a zone-specific baseline established, the system identifies deviations that might be invisible to a global detection system. The nature of these anomalies varies dramatically by industry:
- For a gaming company: A baseline might show millions of users making frequent, rapid API calls to a matchmaking service. A single user making slow, methodical, sequential API calls to scrape the entire player leaderboard would stand out as a clear anomaly.
- For a retail website: Normal traffic flows through a funnel of browsing, cart additions, and checkout. A bot that visits every product page alphabetically at machine speed without ever touching a cart or session cookie is a significant indicator of content scraping.
- For a media publisher: Typical readers follow internal links and spend time on pages. A script hitting thousands of article URLs per minute, spending less than a second on each to extract text for AI training, is an unmistakable anomaly.
In each case, malicious activity is defined not by a universal signature, but by its deviation from the application's unique, established norm.
Step 3: Turning detections into actions
Detection is only useful if it integrates with the security tools already in use. The behavioral detection improvements deliver value in two ways:
- New Bot Detection IDs: Enterprise customers get a new set of IDs that can be used in WAF rules to challenge, rate-limit, or block traffic. Each detection type maps to a unique ID, so customers can see exactly what behavior triggered a flag and filter by Detection ID in Security Analytics for a broader view.
- Improved Bot Score: Requests flagged as anomalous will have their Bot Score lowered, moving them into the "Likely Automated" (scores 2-29) or "Automated" (score 1) categories. Existing WAF custom rules based on Bot Score automatically become more effective without any changes. This is live today for account takeover detections, residential proxy detections, and recent enhancements, and will extend to behavioral scraping detection in the future.
This process is already active for account takeover attacks. Bot Detection ID 201326598, for example, establishes a zone-level baseline, examines anomalous login failures to identify brute force and credential stuffing, and then lets customers mitigate via bot score impact and dedicated analytics.
This integration creates a flywheel effect: intelligence from the new detections strengthens existing products like Super Bot Fight Mode, Bot Management, and the WAF.
First target: sophisticated scrapers
The first generation of the improved behavioral detections is tuned specifically for scraping, analyzing signals beyond simple request headers:
- Behavioral analysis: Examining session traversal paths, request sequences, and interaction (or lack thereof) with dynamic page elements.
- Client fingerprinting: Analyzing automation signals such as JA4 fingerprints within the context of each customer's traffic baseline.
- Content-agnostic detection: Models judge by access patterns rather than page content, keeping the system highly scalable and efficient.
Early testing with closed beta adopters provided ground-truth feedback for tuning. In a single 24-hour period, the new scraping detections flagged 138 million scraping requests across just five beta zones, proving substantial reach. Critically, 34% of those flagged requests would not have been caught by the existing bot score system, demonstrating that behavioral detections provide a genuinely new layer of defense.
Extending protection beyond enterprise plans
The behavioral detections will not be limited to paid bot management customers. Cloudflare is rolling out the enhanced detections to its global Super Bot Fight Mode system as well. Enterprise Bot Management customers get detections automatically tuned to their zone's specific traffic, catching evasive attacks ranging from account takeovers to web scraping to residential proxy abuse.
The infrastructure is designed as a flexible foundation for tackling a wider range of malicious behavior. The same per-customer baseline and anomaly detection principles apply to other application-specific threats, including credential stuffing, inventory hoarding, carding attacks, and API abuse.
The first offering of scraping behavioral detections is near: customers will enable the new detection from the Security Overview page in their dashboard. Generic defenses are no longer adequate; as attacks become more personalized, so must the defenses against them.
