When robots.txt and enforcement disagree

Site owners often maintain multiple layers of bot protection, and those layers don't always agree. A robots.txt file might say a crawler is Disallowed while edge enforcement rules let it through. When stated preferences and enforced rules conflict, some crawlers treat that as license to ignore the preferences or attempt to bypass the enforcement.

Cloudflare's announced Bot Preference Sync addresses this by aligning what you configure for AI bot categories with what gets published to your robots.txt. The feature is available to all customers, from Free tier to Enterprise, and can be enabled or disabled at any time. It builds on earlier work that introduced managed robots.txt values and edge-enforced blocks for AI training crawlers, and extends that approach to Search, Agent, and Training traffic categories.

Different business models, different answers

The questions site owners ask about AI traffic have evolved. Beyond whether content is being used to train models without permission, there are now concerns about discoverability and engagement. How does a site appear when an AI assistant references it? What portion of traffic comes from AI crawlers versus humans? And what is that referral traffic worth?

The answers vary by business model. An e-commerce store may want everything crawled and trained on so products surface in chatbot shopping recommendations. An ad-supported publisher may want to remain in search indexes that drive page views while keeping articles out of model training—and being able to verify that content wasn't used without permission.

Because there's no single right answer, the controls need to reflect each site's strategy. Bot Preference Sync ties the preference you set to the preference you publish, so there's no gap between configuration and the file crawlers actually read.

Transparency as the price of admission

Mixed-use crawlers—bots that combine search, agent, and training behind a single user agent—make it hard for site owners to separate what they want from what they don't. Cloudflare's position on transparency for site owners hasn't changed, but the requirements for bot verification have been extended.

For purposes of bot Verification, owners of bots that perform both Search and Training must now provide additional information to avoid being blocked when "Disallow Training" is set. The requirements are:

  • The bot must respect a "no training" preference in robots.txt via any mechanism.
  • Site owners must have a way to opt out of AI summaries.
  • URL-level visibility into which pages were made available for training, plus metrics on search results, so usage for each purpose is transparent.
  • Public demonstration that Disallowing Training doesn't hurt traditional search results.

Bots meeting these criteria are tracked publicly in the AI bot transparency section of Cloudflare Radar, which also notes when best practices aren't honored. Crawlers that don't provide this transparency remain blocked when training is disallowed.

How Bot Preference Sync works

With Bot Preference Sync enabled, Cloudflare generates or updates your robots.txt based on your AI bot configuration for Search, Agent, and Training categories. The generated content is prepended to any existing robots.txt material, so existing Disallow directives are preserved.

For Search and Agent, the options remain: Allow, Block on pages that serve ads, or Block everywhere. For Training, the Disallow option writes a "no training" preference to robots.txt. Cooperating mixed-use crawlers that meet the transparency requirements can still access content for search indexing, since they allow site owners to verify directly how data is used. Search visibility for those cooperating crawlers is unaffected.

BLOG-3489 2.png
A screenshot of the existing options to configure AI training and search policies, with the new functionality to sync your robots.txt contents with these AI bot policies.

In an example where a site is configured to Allow Search, Allow Agents, and Disallow Training, the prepended robots.txt content would look something like this (shortened for the example):

# BEGIN Cloudflare Bot Preference Sync

User-agent: TrainingBot1
User-agent: TrainingBot2
User-agent: TrainingBot3
User-agent: MixedUseBot-Extended
Disallow: /

...

# END Cloudflare Bot Preference Sync

Cloudflare uses bots tracked in BotBase to periodically update the list of bots written to robots.txt when a category is Blocked or Disallowed. Verified bots classified as Search, Agent, and Training can be viewed in the public bots directory.

Defaults and flexibility

Bot Preference Sync is on by default for all new customers. Existing customers using the legacy managed robots.txt feature will be prompted to review and confirm preferences when the feature launches.

The feature handles category-wide policy decisions rather than case-by-case exceptions. Customers with special arrangements—say, a grant of access to a particular company—can turn off the sync and tailor their file to match custom policies.

One change affects ad-supported publishers specifically. At onboarding, customers can select "I monetize from pages with ads on this domain," which sets Training to Disallow as the default. This keeps content out of model training while preserving search visibility. The setting can be changed at any time. For non-publisher customers, no blocks or disallows are added by default at onboarding—the choice remains with the customer.

Bot Preference Sync will roll out to all customers on every plan in the coming week.