WAF Rule Builder Gets a Natural Language Assistant
One of the most frequent complaints Cloudflare has heard from users is the complexity of crafting custom and rate-limiting rules for its Web Application Firewall (WAF). To address this, the company has introduced an AI Assistant for the WAF Rule Builder. Instead of manually configuring matching criteria, you can now type a command such as "Match requests with low bot score" and the assistant will generate the rule. The goal is not to produce a perfect rule in one step, but to give users a foundation they can refine.
The feature is available in the Custom and Rate Limit Rule Builder for all WAF users and is launching in Beta. Users can provide feedback directly through the UI to help steer the tool's evolution.
AI Bot Traffic Insights Arrive on Cloudflare Radar
AI platform providers often use bots to crawl and scrape websites for model-training data, frequently without the permission of content owners. Alongside the "AIndependence" effort to let sites block these bots, Cloudflare is now offering a view of the activity itself. The new AI bot and crawler traffic graph on Radar's Traffic page shows traffic trends for the top known AI bots over a selected time period. The tracked list is based on the ai.robots.txt list and will expand as new bots are identified.
Developers can also pull this data programmatically through the Radar API, including time series and summary endpoints by user agent. The full set of AI bot and crawler trends is viewable in the new Data Explorer.
Expanded AI Bot Blocking Details
Cloudflare has updated its one-click AI bot blocking feature on the Bots dashboard, following up on the launch of the original button. The earlier version blocked verified AI crawlers that respect robots.txt and identify themselves. The updated rule adds more signatures to that list and also includes detections for 27 AI bots that do not follow these conventions. Cloudflare credits user-submitted tips for helping it identify many of these threats quickly.
For self-service plan customers using Super Bot Fight Mode, these new detections map to the existing "Definitely automated" definition, meaning they are already protected. Enterprise Bot Management customers will also see corresponding shifts in scoring, with more requests classified as definitely automated rather than likely bot.
Under the hood, the rule has been migrated to a Cloudflare managed rule framework, the same one that powers the WAF's managed rules. This allows security analysts to push updates to the rule in real-time, much like rapid CVE mitigation updates. Customers who used the original button should revisit the Bots dashboard and click again to adopt the latest protection.
Model Improvements and Adoption
Fingerprinting these new AI bots has an additional benefit: it improves the supervised machine learning model that underpins Cloudflare's bot management. Introducing heuristics tailored to these bots allows for faster detections and classification with greater certainty than a machine learning approach alone. Rather than relying solely on the complex and time-consuming model deployment process, engineers can create new heuristics in minutes.
Adoption of the original button exceeded 133,000 websites, including significant Free tier usage. The updated version, launched on August 20, 2024, has already been integrated by over 90,000 zones, at a rate of roughly 240 new sites per hour. In total, Cloudflare now protects over 146,000 sites from AI bots and blocks 66 million requests per day with this rule. Support for configuring AI bot protection via Terraform is expected by the end of this year.
Observed Changes in Bot Behavior
Cloudflare has also been monitoring shifts in bot activity following the updates. The daily request volume from AI bots has dropped by an average of nearly 30 million requests over a two-month period, with a decrease of 40 million in the most recent month. This coincides with the releases of both button versions, suggesting that blocking is discouraging some crawlers. The Bytespider bot, for instance, cut its daily requests from roughly 100 million to 50 million between the end of June and the end of August.
There have also been signs of greater accountability among some crawlers. The Perplexity user agent has seen a dramatic surge in requests, which may be linked to previous accusations that the company did not properly present its user agent. Such changes indicate a shift towards more transparent identification and compliance.
Cloudflare will continue to monitor these trends to help users understand and control who accesses their content. The Radar insights update in near real-time, offering a live view of these emerging patterns.
