Granular bot controls now available to every Cloudflare customer
Cloudflare has rolled out new Verified Bot categories that let any user — including those on free plans — block or allow specific types of bots, with AI crawlers as one of the selectable categories. The controls are available through the WAF tab in the Cloudflare dashboard and via Workers, where a rule can reference a bot subcategory as the Field.
The move responds to growing concern among content creators and policymakers about how AI companies collect data for model training. Cloudflare's position is that website operators should have a straightforward way to keep unwanted AI crawlers off their sites while still signaling when crawling is permitted. The company notes that its existing protections — Bot Fight Mode, Super Bot Fight Mode, and Bot Management — already stop malicious scrapers, but the new categories give customers more surgical control over what gets through.
From search engines to AI training data
Web crawlers have been part of the internet since the World Wide Web Wanderer measured the web's size in 1993, and the technique evolved into the search engine indexing that GoogleBot and similar bots perform today. Those crawlers parse HTML, follow links, and store structured data for indexing.
AI crawlers operate similarly — they fetch pages, parse content, and follow URLs — but the harvested content feeds machine learning models rather than search indexes. Cloudflare identifies these bots using the same systems it applies to all crawler traffic: attack signature matching, heuristics, machine learning, and behavioral analysis. The company's Verified Bot directory, previously used to separate responsible bots like GoogleBot into their own detections, now supports the new subcategories.
Ten bot categories, one rule engine
The new categories are:
- Search Engine Crawler
- Aggregator
- AI Crawler
- Page Preview
- Advertising
- Academic Research
- Accessibility
- Feed Fetcher
- Security
- Webhooks
The full set of available categories is also queryable through the Cloudflare API. Targeted responses open up a range of practical use cases:
- Content creators worried about AI reproducing their work can block cataloged AI bots with a simple firewall rule while still letting search engines index their site.
- Sites frequently shared on social platforms can use Workers to serve a simplified page version to Page Preview services such as X (formerly Twitter), Discord, and Slack.
- Online stores processing payments via webhook APIs can restrict endpoints to verified webhook services only, tightening security.
- Cloudflare Load Balancing users with limited regional capacity can route all bots except Search Engine Crawlers to a backup pool, preserving resources for human visitors.
Existing rules that reference bot score or verified bot status are unaffected; customers who prefer no bot traffic need not change anything.
What makes an AI bot respectful
Cloudflare has expanded its Verified Bots policy with criteria for tagging AI bots that operate in good faith. To earn a respectful AI bot tag, an operator must:
- Publish a public page committing to respect robots.txt.
- Use IPs exclusively for the bot, verifiable via public IP list, reverse DNS, or ASN ownership.
- Maintain a unique, stable user-agent.
- Respect robots.txt entries for their user-agent and wildcard entries.
- Honor crawl-delay, a previously nonstandard extension that Cloudflare now requires.
Bot creators meeting those criteria undergo further evaluation: Cloudflare checks documentation, reviews internal dashboards for appropriate traffic distribution across protected sites, and watches for suspicious activity like hits on login endpoints or other malicious behavior.
A broader protocol in the pipeline
Beyond the dashboard controls, Cloudflare is advocating for a new industry-wide exclusion protocol specifically for AI crawlers. The company argues that such a standard would give website operators maximum flexibility over time, letting them decide per page or for their entire site whether AI bots are welcome. Official reporting on bots by category is also planned, which will help customers visualize how frequently different bot types visit their sites.
The protocol details are expected in the coming months; the Verified Bot categories are available now.
