Beyond Crawl Access: A New Way to State Data-Use Preferences

Website operators face a dilemma: keep content openly accessible and risk it being repurposed in unwanted ways—such as for AI training or competing products—or gate content behind logins, limiting reach and undermining the open web. Today’s tools only control access at the point of crawling, not what happens after data is fetched. Cloudflare’s new Content Signals Policy introduces a standard, machine-readable method for content creators to express their preferences for post-access usage, extending the existing robots.txt convention.

The Limits of robots.txt

The Robots Exclusion Protocol, implemented via a plain-text robots.txt file on a domain, tells crawlers which user agents may access which parts of a site. Many bots obey these instructions, but not all. For instance, a simple directive allows all crawlers to access the entire site:

User-agent: * 
Allow: /

The * user agent matches any visitor—bot or browser—and the / in Allow permits access to all paths. Comments, introduced by a # character, offer a human-readable layer that bots ignore. More specific rules can restrict certain crawlers or directories:

User-agent: * 
Disallow: /archives/

These directives govern what can be crawled and by whom. They say nothing, however, about how fetched content may be used afterward. As the industry has recognized, a standard for signaling usage rights after access has been missing. The Content Signals Policy addresses this gap.

The Impetus Behind Content Signals

The current web economy is strained by large-scale data scraping. Serving these requests costs operators money with little return, a classic free-rider problem. Projections show bot traffic increasingly dominating the web, with bot-only activity expected to exceed the sum of today’s total Internet traffic by 2031.

Historically, the implicit bargain was that scraped content yielded referral traffic or attribution. The practice of linkbacks, for example, gave creators credit without direct payment but drove future discovery. That norm, embedded in permissive licenses such as the MIT License and Creative Commons (which require attribution), has eroded. Scraped content is now sometimes used to compete economically with the original creator, leaving operators the choice of locking down content or accepting minimal acknowledgment.

How the Content Signals Policy Works

The policy is integrated into robots.txt as commented, human-readable text that also defines machine-readable signals. It introduces a vocabulary of three content signals—search, ai-input, and ai-train—relating to how crawlers may use accessed content.

# As a condition of accessing this website, you agree to abide by the following content signals:

# (a)  If a content-signal = yes, you may collect content for the corresponding use.
# (b)  If a content-signal = no, you may not collect content for the corresponding use.
# (c)  If the website operator does not include a content signal for a corresponding use, the website operator neither grants nor restricts permission via content signal with respect to the corresponding use.

# The content signals and their meanings are: 

# search: building a search index and providing search results (e.g., returning hyperlinks and short excerpts from your website's contents).  Search does not include providing AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g., retrieval augmented generation, grounding, or other real-time taking of content for generative AI search answers). 
# ai-train: training or fine-tuning AI models.

# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.

The policy’s structure has three components:

  • Interpretation rules: Signals express preferences with “yes” or “no.” The absence of a signal carries no meaning—this neutrality allows an operator to take a stance on one usage without being forced to declare a position on another.
  • Signal definitions: The vocabulary is deliberately simple to ease compliance by automated agents.
  • Legal note: A reminder that content signals may have legal implications in some jurisdictions.

To use these signals, operators append comma-delimited yes/no preferences to their robots.txt. A config allowing search, blocking ai-train, but saying nothing about ai-input would not mean consent is given for the latter; rather, the operator has simply not expressed a preference there.

User-Agent: *
Content-Signal: search=yes, ai-train=no 
Allow: /

Deployment for Cloudflare Customers

For the over 3.8 million domains already using Cloudflare’s managed robots.txt feature—which signals that content should not be used for AI training—the file Cloudflare serves on their behalf will be updated to include the Content Signals Policy and a directive against ai-train:

Content-Signal: search=yes, ai-train=no

An ai-input signal will not be served automatically for these customers, as Cloudflare cannot know their preference on that matter.

Starting immediately, free-plan zones without an existing robots.txt file will receive the commented, human-readable Content Signals Policy in response to robots.txt requests. This serves as an informational header only; it contains no Allow or Disallow directives, and no actual content signals are applied. Site owners may choose to express their preferences themselves. Existing robots.txt files will be unaltered. Free-zone users can disable this feature via the Security Settings section of the dashboard or the Overview section.

Content signals are preferences, not enforcement mechanisms. Some companies may disregard them. For stronger protection, operators should pair them with WAF rules and Bot Management tools. To promote open adoption, the policy itself is released under a CC0 License, permitting free implementation and use.

Looking ahead, success hinges on broader recognition and standardization of these signals. Cloudflare plans to continue working with standards bodies to develop accepted solutions, aiming to preserve an open web where creators retain control over their work’s downstream use. The tools at ContentSignals.org allow anyone to generate the necessary text for their robots.txt, with options for immediate deployment.