Your site, your rules: new AI traffic options for all customers

A Year On, AI Traffic Controls Get More Granular

A year after Cloudflare first gave website owners a one-click way to block AI crawlers, the company is expanding those controls significantly. The original premise was a straightforward win-lose deal: AI companies were taking content for model training without sending back any traffic. The new approach acknowledges that not all automated traffic is the same, and that locking everything down isn’t the right answer for every site.

The challenge is particularly acute for smaller publishers. They face a difficult choice between being visible to AI-driven search engines and protecting their content from being absorbed into models. The update aims to give those site owners more nuanced options than an all-or-nothing block.

Moving Beyond the 'AI Bot' Label

The definition of what constitutes "AI" is constantly shifting, making it a poor basis for classification. Cloudflare is moving away from that label toward a behavioral taxonomy that focuses on what a bot is actually doing on a site. The new system centers on three core use cases for AI traffic:

• Search: Behavior that collects or indexes content to build a database for answering queries later. This is typically proactive crawling, and site owners should expect referral traffic in return.
• Agent: Automated behavior acting in real time on a person's behalf to complete a task. This includes chat fetch bots (like ChatGPT-User) and browser-use agents (like Gemini or Claude driving Chrome). A human is often waiting on the other end for the job to be done.
• Training: Crawling to train or fine-tune a model, where data is permanently absorbed into the architecture of the AI.

A single bot can fall into multiple categories. The company is strongly encouraging bot operators to separate their automation into distinct crawlers for each purpose. This creates more transparency, allowing site owners to understand why a crawler is visiting and to apply the appropriate rules.

New Controls and New Defaults

The existing "Block AI bots" preset in zones will be replaced with three separate toggles for Search, Agent, and Traffic crawlers. These options will be available to all customers, including those on the Free tier.

New defaults will be applied on September 15 for all domains onboarding to Cloudflare. On pages that display ads, Training and Agent crawlers will be blocked by default, while Search will remain allowed. The logic is that an ad is a signal the owner wants human attention, and allowing bots that don't funnel visitors back is counterproductive.

Also starting on that date, the handling of multi-purpose crawlers will change. They will be governed by all of their behaviors, not just their most lenient one. This means that crawlers combining Search with Training—such as Googlebot, Applebot, and BingBot—will be blocked if a customer has chosen to block Training. Customers who prefer the current behavior can opt out of these new defaults in their Security settings ahead of the change.

BotBase: A New Directory for Enterprise

For Enterprise Bot Management customers, a new feature called BotBase provides a comprehensive, searchable view of all known Verified bots and agents directly in the dashboard. It offers a filtered look at the entire catalogue and how each bot is classified under the new taxonomy. For precise targeting, users can filter all traffic from a specific bot or copy its detection ID for use in security rules.

BotBase is being built to track a bot's content use—what it keeps and reshare after crawling. This is classified into one of three levels from least to most permissive:

• immediate — interact, but store and reuse nothing
• reference — index, excerpt, and link back
• full — summarize and reproduce

These levels can be combined with bot classifications to create nuanced rules. For example, an owner could allow all bots used for Search but only up to the reference use level. The taxonomy also includes well-known groupings that are not configurable, offering further context on bot behavior.

Extending Content Signals

To support this preference, the Content Signals specification is being extended with a new, optional use field that can be added to robots.txt. Like other directives, this is a stated preference rather than an enforcement, and values can be use=immediate, use=reference, or use=full.

# Cloudflare Managed content with original Content Signals

User-agent: *
Content-Signal: search=yes,ai-train=no
Allow: /

# Cloudflare Managed content with the new content-use signal

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

Customers already using Cloudflare's managed robots.txt—which signals that search crawling is okay but training is not—will see the use=reference line automatically added to their file. Cloudflare will also begin tracking content uses for every bot in BotBase. Bots that reproduce content in full are not currently eligible for Verified status.

Rethinking 'Verified' Status and Trust

The definition of what it means to be a "Verified" bot is changing. Previously, all Verified bots were allowed by default. Now, the Verified label makes a bot allowable by its category, meaning the owner's choice of allowed categories determines access. The process for becoming Verified is being opened up, requiring operators to show they represent themselves honestly and don't abuse the access that earns them.

A related concept being explored is transitive trust. This applies when a bot operator isn't the company that built the bot. To handle trust across intermediary layers, Cloudflare is leveraging the existing Forwarded header from RFC 7239, which is also being extended with the content-use field:

Forwarded: for="openai";use="reference"

This allows an owner's preference for a specific operator to hold, even if traffic comes through multiple trusted layers. The system has its limits, particularly for users who can't afford to be identifiable or for companies with strict privacy commitments, but it is a step toward an agentic internet built on verifiable signals.

The new AI traffic controls, searchable BotBase directory, and updated settings are now available to configure in zone settings. The new defaults will be set on September 15, 2026.