AI Crawl Control gives site owners visibility and enforcement
Until now, website owners had almost no way to see how AI services were using their content, whether for model training, search indexing, or archiving. Cloudflare’s new AI Crawl Control feature changes that. Every site on Cloudflare—on any plan, with no configuration required—now gets a detailed analytics view of AI bot traffic plus one-click blocking and granular filtering tools.
Three categories of bots have long existed: good bots like search crawlers that drive traffic, bad bots that attack or scrape, and now a murkier third group tied to AI platforms. These AI-related crawlers are not malicious in the traditional sense; they scan public content without attempting to break anything. But unlike search engines, they often send no traffic back. AI data scrapers fold your material into training sets that answer queries without attribution. AI search crawlers may cite sources but keep users inside their own interface, so visits never reach your site.
That ambiguity has left publishers with an unclear value exchange and few options beyond blocking everything or allowing unrestricted access. Cloudflare’s release introduces a five-step framework for taking control.
Step 1: See which AI services are crawling your site
In the Cloudflare dashboard, select a site and open the AI Crawl Control (formerly AI Audit) tab. The view summarizes crawl activity from known AI services, broken down by provider, bot type, and the specific pages being accessed.
Bots typically identify themselves through a User Agent HTTP header, though Cloudflare may also rely on IP address or behavioral heuristics when the header is absent. For instance, Anthropic’s crawler sends ClaudeBot, while OpenAI uses different agents for different purposes: GPTBot for data scraping and OAI-SearchBot for its search engine. That distinction matters—a search crawler may drive referral traffic, while a data scraper simply consumes content.
Filters let you review activity by category—AI Data Scraper, AI Search Crawler, and Archiver—so you can understand not just how often bots visit, but why.
Step 2: Use the pause button
If the data is overwhelming and your team has not yet settled on a policy, Cloudflare provides a one-click way to buy time. In the Security tab under Bots, toggle the “Block AI Scrapers and Crawlers” card to “On.” This applies Cloudflare’s maintained list of known AI bots and blocks them all immediately, letting you decide deliberately what to allow later.
Step 3: Enforce selective policies
Once your team decides, the WAF section of the dashboard supports rules that go beyond all-or-nothing. You can block all AI bots except those from specific providers, or allow only certain bot types—for example, permitting AI search crawlers while blocking data scrapers. Site administrators who have signed agreements with a single AI partner can create a rule that blocks everything and then adds an exception for that partner’s specific bots.
Cloudflare also recommends updating your site’s Terms of Service to define that AI scanning must respect the policies in your robots.txt file. Sample language is available in Cloudflare’s developer documentation.
Step 4: Audit existing direct deals
Publishers who have negotiated licensing agreements with AI companies often need to verify compliance with contractual terms about crawl frequency and content access. The AI Crawl Control table lists your most popular content by scan count for the selected time period. The Export to CSV button downloads the full details—covering per-page requests and total site requests—so you can compare actual crawling behavior against what your agreement permits.
Step 5: Prepare for monetization
Today only the largest publishers have the resources to negotiate paid licensing arrangements directly with AI providers. Cloudflare is previewing a monetization component that aims to change that. Site owners will be able to set prices for their entire site or for specific sections, then charge model providers based on scan volume. Cloudflare handles the rest.
The beta will be offered to sites based on when they first joined Cloudflare. Site owners can join a waitlist to be notified when the feature becomes available.
