AI Crawl Control Leaves Beta
Cloudflare has rebranded its AI Audit offering as AI Crawl Control and moved the product to general availability. The shift is more than cosmetic: AI Audit was designed for passive observation, while AI Crawl Control pairs those insights with active levers for dictating how AI systems interact with a site's content. The company initially announced its "Content Independence Day" framework in July, and this release builds on that foundation.
The core pain point the tool addresses is familiar to publishers balancing crawl traffic against business interests. Blocking all AI bots can cut off referral traffic and potential licensing conversations, while letting every scraper through leaves content exposed without compensation. AI Crawl Control sits between those extremes, letting site owners fine-tune access on a per-bot basis.
HTTP 402 for Licensing Conversations
A central element of the update is broader access to customizable HTTP 402 ("Payment Required") response codes. Previously, this capability was limited to participants in Cloudflare's private beta for Pay Per Crawl, which paired the 402 mechanism with its Web Bot Auth feature for direct payments between agents and publishers. Now, every paying Cloudflare customer can send custom 402 responses through the AI Crawl Control dashboard.
The workflow is straightforward: an admin selects a specific crawler that they want to block and toggles the response type to 402. The status code is shipped with a configured message containing any instructions the site operator wants to attach. The intended result is a channel for conveying terms before a crawl begins, rather than an outright refusal.
Typical message templates the company suggests include:
- "To access this content, email [email protected] or call 1-800-LICENSE"
- "Premium content available via API at api.yoursite.com/pricing"
The response handling works like this:
Messages are configured from the Settings menu in the AI Crawl Control dashboard:
Early Signal and Roadmap
Cloudflare reports that its customers are already returning over one billion 402 response codes on an average day. That volume, the company says, points to demand for a middle path between open access and total shutdown — a way for site owners to initiate a commercial conversation instead of just closing the door.
Future iterations are planned to enrich the response with structured metadata. Instead of only contact details, the 402 payload would expand to include information about the content's value, its freshness, or update cadence, giving crawlers more context with which to make access decisions.
The separate Pay Per Crawl beta also continues. It is designed to give creators infrastructure for automated monetization of crawler access based on usage, or "pay per crawl." Taken together, the two efforts point to an ecosystem where licensing terms and usage pricing are built into the crawling protocol itself, rather than negotiated afterward through back channels.
