One-click blocking for AI scrapers and crawlers

Cloudflare has introduced a new one-click control that lets website operators block all AI bots, crawlers, and scrapers. The feature, labeled AI Scrapers and Crawlers, is available to all customers, including free-tier users, and can be enabled from the Security > Bots section of the Cloudflare dashboard.

The move comes as demand for bulk content — used to train large language models or power retrieval-augmented generation systems — has surged. While some AI companies identify their crawlers and follow robots.txt conventions, others have been accused of disguising their traffic. Cloudflare reports that even when AI bots behave well and comply with the rules, its customers overwhelmingly choose to block them.

Who is crawling the web?

Cloudflare analyzed traffic across its network over the past year to identify the most active AI crawlers. In terms of request volume, the top four are Bytespider, Amazonbot, ClaudeBot, and GPTBot.

Bytespider, operated by ByteDance (the parent company of TikTok), reportedly gathers training data for its large language models, including those powering its ChatGPT rival Doubao. It leads not only in raw request volume but also in the breadth of Internet properties it accesses and the frequency with which it is blocked. Amazonbot, reportedly used for Alexa’s question-answering, ranks second in request volume, followed by ClaudeBot, which has seen recent increases in activity. GPTBot, managed by OpenAI, ranks second in both the extent of crawling and how often it is blocked.

Despite the activity of these major crawlers, analysis of the top 10,000 domains on the Internet shows that website operators most often reference GPTBot, CCBot, and Google in their robots.txt files — while less frequently disallowing other highly active crawlers like Bytespider and ClaudeBot.

How website operators are responding

Cloudflare observed that in June, AI bots accessed roughly 39% of the top one million Internet properties protected by Cloudflare. However, only 2.98% of those properties took measures to block or challenge those requests. The data also shows that the more popular a website, the more likely it is to be targeted by AI bots — and the more likely it is to respond by blocking them.

robots.txt remains a common defense, but it depends on bot operators honoring the file and honestly identifying themselves as required by RFC9309. User agents are trivial to change, and Cloudflare notes that bot operators have been observed doing exactly that.

Identifying bots that lie about their identity

Cloudflare’s global machine learning model has consistently classified traffic from evasive AI bots as bot activity, even when those bots spoof user agents to appear like real browsers. In one example — a bot others have accused of hiding its activity — the company ran an analysis of its bot scores and found that the vast majority of requests scored 9 or below on a scale where lower scores indicate bot traffic. The user agent changes had no effect on the score.

Customers with WAF rules set to challenge visitors with a bot score below 30 (Cloudflare’s recommendation) automatically blocked this traffic without any additional configuration. The same approach applies to future AI bots that adopt similar evasion tactics.

Cloudflare says it is able to detect these bots by fingerprinting the tools and frameworks used for large-scale crawling, then cross-referencing that against global signals from its network, which handles an average of 57 million requests per second. This allows the models to flag new scraping tools and behaviors without requiring manual fingerprinting for each new bot.

Reporting misbehaving AI crawlers

Cloudflare offers two ways to report AI bots that are scraping without permission:

  • Enterprise Bot Management customers can submit a False Negative Feedback Loop report via Bot Analytics by selecting the segment of traffic where misbehavior was observed.
  • Any Cloudflare customer can use the company’s reporting tool to submit reports of AI bots scraping their website without permission.

Cloudflare says the AI Scrapers and Crawlers blocklist will be updated automatically over time as new offending bots are identified. The company intends to continue expanding its machine learning models to help content creators retain control over whether their material is used for AI training or inference.