Why the Internet needs rules for AI bots
Cloudflare occupies a unique position on the Internet: we observe not only how technology shifts reshape the web, but also how those shifts create unintended consequences for different players. The rapid adoption of AI-powered chatbots and search summaries is a prime example. End users get answers faster, but publishers who depend on human visits to fund original content are seeing a dramatic drop in traffic. That loss of revenue undermines the very content supply that AI companies rely on for training and service quality — a cycle that ultimately hurts everyone involved.
The Internet is clearly at a point where AI bot behavior needs explicit ground rules. Through ongoing conversations with stakeholders, publishers, and partners, it has become evident that a consensus is needed among three key groups: content creators, bot operators, and infrastructure or cybersecurity companies. Reaching agreement on specific principles will take time, and no policy framework can capture every technical nuance. But starting the conversation is essential — a rough draft beats a blank page.
To that end, we propose the following responsible AI bot principles as a foundation for discussion:
These principles focus on AI bots because of generative AI's current impact on the Internet, though we have seen similar practices work with non-AI bots too. We believe they point the web in a better direction, but they are only a starting point. Input from other stakeholders will be necessary, and these principles should evolve as the Internet itself does.
Why Cloudflare is pushing this conversation forward
Since declaring July 1st Content Independence Day, Cloudflare has worked to play a balanced role in safeguarding the Internet's future in the age of generative AI. We have given customers the ability to charge AI crawlers for access or block them with one click, published and enforced our verified bots policy, developed the Web Bot Auth proposal, and called out deceptive crawler behavior when we have seen it.
This work builds on a longer history in bot management, helping customers defend against unwanted and malicious traffic. We also maintain a public view — for anyone, not just our customers — of which AI bots follow all, some, or none of these best practices. At the same time, we recognize that bot operators are adapting to a new landscape themselves. Most players in this space want to do the right thing while continuing to innovate. Our hope is that our vantage point can help bring seemingly incompatible parties together, in line with our broader mission of helping build a better Internet.
The five responsible AI bot principles
These principles are a launchpad for a larger conversation, and we acknowledge that many nuanced perspectives remain to be addressed. We envision them applying to AI bots, but technical complexity may require flexibility. Our ultimate goal is transparency, accountability, and respect for content access and use preferences. If these principles fall short — or miss other important priorities — we want to hear about it.
Principle 1: Public disclosure
Companies should publicly disclose information about their AI bots. The following should be easy to find and publicly available:
- Identity: information that helps external parties identify a bot, such as user agent, relevant IP addresses, or individual cryptographic identification (see Principle 2).
- Operator: the legal entity responsible for the bot, including a point of contact for reporting abuse.
- Purpose: what the accessed data will be used for — search, AI input, or training (see Principle 3).
OpenAI is a leading example of clear disclosure, publishing detailed explanations of each bot's purpose. This makes it straightforward for website operators to verify that a request is genuinely from OpenAI and to understand its intent, such as search indexing versus model training. That, in turn, enables site owners to control access through mechanisms like robots.txt files.
Principle 2: Self-identification
AI bots should truthfully self-identify. Beyond public documentation, bots should clearly communicate who they are in their requests — for example, through an HTTP request carrying an official user agent and originating from an IP address the bot has claimed. This approach is admittedly imperfect, as we discuss below, but until cryptographic verification becomes widespread, user agent and IP verification is better than nothing.
OpenAI's GPTBot illustrates this principle. OpenAI publicly shares the full expected user-agent string, explains that the bot is used to make generative AI foundation models more useful and safe and to crawl content for training, and we have observed traffic coming from the IP addresses OpenAI reports. Because site operators can match the bot's user agent and IP addresses against public documentation, they can confidently identify GPTBot and make informed decisions about whether to allow its traffic.
Not all bots uphold this standard. Anthropic publishes only its user agent, with no additional verifiable information, leaving it unclear which requests actually come from Anthropic. xAI's grok bot does not self-identify at all, making it impossible for website operators to block it. Both cases undermine trust, and both could be fixed with minimal effort from the operators.
Beyond user-agent lists: cryptographic verification
Traditionally, Principle #2 compliance has been demonstrated by declaring a user agent and publishing dedicated IP lists. That approach is increasingly fragile. Bad actors spoof user agents, and robots.txt only expresses preferences on a user-agent basis—so a malicious bot can impersonate a permitted crawler and bypass a site's directives entirely.
The long-term fix is cryptographic verification. Every AI bot should be verifiable through an accepted standard, preventing spoofing and giving website operators reliable information about who is accessing their content. At this stage, we consider Web Bot Auth sufficient proof of compliance with Principle #2, though we recognize the standard is still in development and this principle may evolve accordingly.
Web Bot Auth uses cryptography to verify bot traffic by attaching cryptographic signatures to HTTP messages. Our implementation relies on proposed IETF directory and protocol drafts. Adoption is gaining momentum: Vercel announced support for Web Bot Auth in its bot verification, and OpenAI's ChatGPT agent now signs its requests using Web Bot Auth alongside the HTTP Message Signatures standard. We envision a future where cryptographic authentication is the norm, strengthening the trustworthiness of all bots.
Principle #3: Declared single purpose
AI bots should have one distinct purpose and declare it. Some bots currently self-identify as Training, Search, or User Action—the last meaning accessing content in response to a user's query. But these purposes are often combined without clear distinction. Content fetched for search might also train the model powering that search engine. When purpose is ambiguous, operators face an impossible choice: block the bot and risk SEO, or allow it and risk unwanted content use.
Distinct, declared purposes let website owners make deliberate choices. Based on traffic we observe, we propose this starting taxonomy, recognizing it will evolve:
- Search: building a search index and providing results—returning hyperlinks and short excerpts. This does not include AI-generated search summaries.
- AI-input: feeding content into AI models—for example, retrieval-augmented generation (RAG), grounding, or real-time content retrieval for generative search answers.
- Training: training or fine-tuning AI models.
Bots should not combine purposes in ways that prevent operators from deliberately deciding whether to allow crawling.
Consider two bots from the perspective of Vinny, a website operator trying to earn a living. OAI-SearchBot declares a single purpose: linking to and surfacing sites in ChatGPT's search features. Vinny can trust it won't crawl his content for training—a separate bot, GPTBot, handles that—and can allow search access while denying training, confident his choices are honored. Googlebot, however, scrapes for traditional indexing and uses that content for inference in AI Overviews and AI Mode. Vinny wants his content in search results—those drive the eyeballs that fund his site—but he's forced to accept AI-generated summaries too. If a summary satisfies the user, no one visits his site, fueling "zero-click" searches that undermine his revenue.
This creates a vicious cycle: high-quality content ranks better, but that same content gets surfaced in AI Overviews, reducing visits. Vinny's only options are opting out of search entirely or degrading his listings with snippet controls—because Google-Extended only disallows training, not AI Overviews attached to search.
The common argument for combining purposes—that it reduces crawler costs—doesn't hold up. It's like claiming ordering two pizzas in one phone call is cheaper than ordering them in two calls. The pizzas cost the same either way; the extra call is trivial. Similarly, whether one bot request carries two purposes or two separate requests each carry one, the compute costs are identical—content must be processed for each purpose—and the marginal cost of an additional connection is negligible. This is not hypothetical: Cloudflare handles roughly 84 million requests per second, and we understand request economics at Internet scale. (And if additional crawls do impose costs on operators, they can choose whether the crawl is worth it when purposes are distinct.)
Principle #4: Respect preferences
AI bots should respect preferences expressed by website operators where proportionate and technically feasible. Operators have several options for expressing these preferences, from the longstanding robots.txt standard to emerging HTTP headers.
Bots should make a good-faith attempt to fetch robots.txt first, per RFC 9309, and abide by both access and use preferences. Operators should also track an IETF draft vocabulary now under development, which aims to add granularity to robots.txt so site owners have finer-grained control over asset use.
Additionally, emerging standards may attach machine-readable preferences to individual files. AI bot operators should prepare to comply with these as well—including ideas like expressing preferences via HTTP headers, which offer a server-level method of declaring content usage.
Principle #5: Act with good intent
AI bots must not flood sites with excessive traffic or use deceptive behavior. Their actions should benefit website operators and users. Bot operators are responsible for monitoring their networks, patching vulnerabilities, and never jeopardizing a site's security or performance.
Nor is it acceptable to appear compliant while secretly circumventing the rules. Reaffirming a long-standing principle of bot behavior, AI bots must never engage in stealth crawling or evasion tactics—modifying user agents, shifting ASNs to hide activity, or ignoring robots.txt. Such behavior undermines the other principles, harms operators, and degrades the Internet for everyone.
Putting the principles into practice
As we refine these principles, we're balancing content creators' wishes against AI innovation. We continue to engage right holders, AI companies, policy-makers, and regulators to shape global standards and frameworks. Generative AI need not threaten the open web's role as a source of quality content—but protecting that role requires agreed technical standards reflecting the interests of publishers, creators, and AI companies alike.
Neutral forums like the IETF are essential—current work there addresses some technical aspects, but not all issues in a space needing holistic consideration. Our principles are a step in the right direction. We hope others join this complex conversation so Internet norms can adapt to this new era.
