Monetized pages get an AI-crawler moat
Cloudflare is rolling out two new controls designed to give site owners more direct authority over how AI bots interact with their content, with an emphasis on protecting ad-supported publishers. The first lets Cloudflare handle the creation and upkeep of a site’s robots.txt file, automatically generating the appropriate directives to exclude AI training crawlers. The second is an option to block AI bots specifically on sections of a site where monetization happens through ads, leaving the rest of the site accessible to general traffic.
These features arrive against a backdrop of shifting crawler economics. Traditional search engines like Google have maintained a mutually beneficial arrangement with publishers: their bots crawl content to index it, and in return they send referral traffic that supports ad revenue. The ratio of crawls to referrals for Google sites sits at roughly 14:1 as of June 2025. But newer AI training crawlers have upended that trade. According to Cloudflare Radar, that ratio is substantially worse for major AI vendors. In June 2025, OpenAI's crawl-to-referral ratio was measured at 1,700:1, and Anthropic's at 73,000:1.
The distinction is not simply a technical matter. Search crawlers such as Googlebot have historically driven more viewers to a site. AI training bots, while technically identical in operation, consume content to serve answers for their own users, and typically return almost nothing in terms of referral traffic. Cloudflare notes that its measurement is an estimate, particularly since referrals coming from native apps may not include a Referer: header, but the asymmetry is clear enough to threaten the viability of independent publishers that hinge on visitor numbers.
Despite this imbalance, coverage of basic blocking measures remains low. Analysis shows that only about 37% of the top 10,000 domains on the internet currently publish any robots.txt file, indicating a widespread underuse of one of the simplest tools to govern crawler access. Cloudflare's broader push has been to lower that barrier, emphasizing a belief that a wider ecosystem of independent publishers is a key condition for a healthy internet.
How the crawler landscape has shifted
Cloudflare first offered a free "one-click" tool to block AI scrapers in July 2024. Since then, over a million customers have enabled that feature, producing a sizable set of observations about AI bot behavior.
The most notable change has involved previous traffic leaders. Bytespider, which held the top spot in Cloudflare's initial assessment, has seen its traffic volume drop by 71.45% since the first week of July 2024. The percentage of Cloudflare-protected sites it visits has also fallen markedly, from over 40% to just 9.37%. In contrast, GPTBot has surged in request volume, now exceeding the traffic generated by large traditional tech players such as Amazon and ByteDance. However, GPTBot now reaches fewer sites than it previously did; its reach among Cloudflare-protected sites has declined from 35.46% to 28.97%, a sign that customer blocking is having an effect.
Activity from AI Search and AI Assistant crawlers has grown rapidly over the last six months. Yet their total traffic is still marginal compared to the load from AI training bots, which went up by 65% during that same period. This sustained growth underscores how profitable content acquisition remains for model builders despite the technical and legal friction.
To help owners identify and act on these trends, Cloudflare introduced free granular auditing in September 2024. This gave sites visibility into which crawlers were accessing their content most often, along with the option to block specific bots via simple templates. By December 2024, Cloudflare had added an automated method to block crawlers found to be ignoring the directives in robots.txt.
Managed robots.txt: putting guardrails on AI crawlers
A website’s robots.txt file has long served as the de facto code of conduct for search engine crawlers. It spells out which parts of a site bots may access, using directives like User-agent, Disallow, Allow, and Sitemap. The IETF formalized the syntax and precedence rules in RFC 9309, which also stresses that compliance is voluntary, not enforced access control. Well-behaved crawlers follow the file; misbehaving ones risk being banned.
AI crawlers have changed the stakes. Very few of the top domains on Cloudflare even publish a robots.txt file, and among those that do, AI-specific user agents are rarely disallowed. As of publication, GPTBot is disallowed in only 7.8% of robots.txt files found on the top 10,000 domains; Google-Extended appears in just 5.6%, while anthropic-ai, PerplexityBot, ClaudeBot, and Bytespider each show up in under 5%. The gap between GPTBot disallow rates and those for other major AI crawlers suggests a widespread misconfiguration: blocking GPTBot alone does nothing to stop the others, and many site owners overlook critical user agents like Google-Extended even when they intend to keep content out of AI training.
Signals in the data
Cloudflare Radar tracks how the top domains handle AI crawlers in robots.txt and publishes weekly snapshots. Comparing June 2025 with January 2025 of the same year, two trends stand out. Very few new robots.txt files appeared among top domains in that six-month window. More striking, however, is the shift in permission types: the proportion of sites marking AI crawlers as “Fully Disallowed” rose sharply, while “Partially Disallowed” declined. Site owners are moving from nuanced rules to blanket blocks on top AI bots including GPTBot, CCBot, and Google-Extended.
Cloudflare’s managed robots.txt
Website owners face a genuine tension. Allowing Googlebot is essential for search visibility, but that same crawler is used for both SEO indexing and AI training. The way to distinguish between the two is by disallowing Google-Extended in robots.txt, leaving Googlebot untouched for standard search crawls.
Cloudflare’s managed robots.txt feature automates that distinction. Once enabled, Cloudflare generates or updates a robots.txt file for a zone with directives requesting that popular AI bot operators refrain from using the site’s content for model training. The managed file targets user agents including Google-Extended and Applebot-Extended while leaving the domain’s SEO configuration intact. The feature is available to all customers and can be turned on from the Cloudflare dashboard.
For sites that already maintain their own robots.txt, Cloudflare prepends its managed directives to the existing file rather than overwriting it. That preserves any custom blocking rules while layering on AI crawler guardrails. Cloudflare also plans to keep the managed directive set current as new AI crawlers emerge, so customers do not need to track the landscape themselves. New sites onboarding to Cloudflare will be prompted with options for handling AI crawlers at setup.
Technical implementation
Under the hood, Cloudflare added a new module that intercepts every inbound HTTP request for /robots.txt. For each such request, the module checks a distributed key-value store to see whether the zone has opted into the managed file. When enabled, the module responds with the managed directives, prepended to the origin’s file if one exists.
Prepending is deliberate. The generalized header is based on the IETF AI preferences proposal, and robots.txt precedence rules require the most specific match to win. Since Cloudflare’s disallow expressions are intentionally scoped to cover everything, a prepended directive cannot conflict with a more targeted customer rule later in the file. When the feature is disabled, requests are forwarded to the origin untouched.
Caching the origin’s robots.txt was considered to cut round trips, but Cloudflare decided against it. Robots.txt requests make up a tiny fraction of total traffic, so the latency savings would be negligible, and the added cache invalidation complexity was not worth the marginal benefit.
Covering the blind spots
Managed robots.txt closes a gap between intent and configuration. Manually maintaining an up-to-date robots.txt that accounts for every new AI crawler is time-consuming, and the Radar data shows that few site owners get it right. Cloudflare’s approach keeps the file current, preserves custom customer rules, and requires no ongoing action once enabled — all while leaving SEO-affecting crawlers like Googlebot untouched.
Targeting the block where ads actually run
A blanket robots.txt entry is only an honor-system request, and nothing forces a bot to respect it. Cloudflare's one-click managed rule goes further by blocking AI bots across an entire zone. But many operators want AI crawlers to reach benign pages, such as developer docs or support articles, while keeping them away from monetized content. A new option handles that case: Cloudflare detects when a hostname serves ads and blocks AI bots only on that hostname.
The first step is recognizing AI bot traffic. Well-behaved crawlers that publish their user agents and use dedicated IP ranges are often added to Cloudflare's Verified Bot list, sometimes working directly with the bot makers. Some operators go further and cryptographically verify their requests with Cloudflare. Less scrupulous crawlers spoof user agents to look like real browsers, but Cloudflare's machine learning models are trained to flag this activity using fingerprints of the tools and frameworks commonly used for large-scale crawling. The network's average of over 57 million requests per second provides a baseline for trust signals, and global aggregates of those signals let the models consistently flag evasive AI bots.
Once a request is identified as an AI bot, Cloudflare checks whether the target page actually serves ads. The inspection happens on the response body — the raw HTML being returned. After parsing the document, the system scans for code patterns typical of ad units, such as:
<div class="ui-advert" data-role="advert-unit" data-testid="advert-unit" data-ad-format="takeover" data-type="" data-label="" style="">
<script>
....
</script>
</div>A div container with the ui-advert class is a common advertising marker. Links to well-known ad servers like Google Syndication also provide a strong signal:
<link rel="dns-prefetch" href="https://pagead2.googlesyndication.com/">
<script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-1234567890123456" crossorigin="anonymous"></script>The scan uses Cloudflare's LOL HTML parser, which streams and parses small chunks of the response directly. This keeps the inspection from adding latency to the served response.
The detection approach borrows techniques that ad blockers have used for years. Ad blockers perform two tasks: blocking browser requests to ad servers and suppressing HTML elements containing ads. They rely on large filter lists, such as EasyList, containing URL block filters that match outbound request URLs against patterns, and CSS selectors designed to match HTML ad elements. Cloudflare applies both techniques — checking external resources referenced by HREF or SCRIPT tags against URL filters, and matching HTML elements against CSS selectors. Because the goal is detecting the presence of ads rather than blocking every individual ad, the filter set can be trimmed from EasyList's 40,000+ rules down to roughly 400 of the most commonly seen ones, improving computational efficiency.
Some sites load ads dynamically rather than embedding them directly in the initial HTML, partly to evade ad blockers. To cover those cases, Cloudflare supplements the HTML scan with data from Content Security Policy (CSP) reports. CSP is a security standard that lets developers control which resources a browser may load for a page, and browsers send reports on loaded resources to a management system. For many sites, Cloudflare's Page Shield product fills that role. These reports connect scripts loaded from ad servers back to the page URLs that loaded them. An endpoint management service consumes both information sources — the HTML scans and the CSP reports — and matches incoming requests against hostnames already known to be serving ads.
Every customer who opts in gets this check on every request, including those on free plans. Enabling it requires navigating to the Security > Settings > Bots section of the Cloudflare dashboard and choosing either Block on pages with Ads or Block Everywhere.
How AI bot categories get built
The population of AI crawlers and scrapers keeps expanding as new operators come online. Cloudflare's security research team continuously identifies and classifies these agents across the network's traffic. Two primary channels feed that classification effort.
Customers submit reports about misbehaved AI bots that Cloudflare has not yet classified, using the bots submission form. Those reports send security analysts to investigate and decide how the bot should be categorized. Separately, Cloudflare analyzes the large scale of observed customer traffic to determine which AI agents visit which websites and when, revealing patterns that might make a website owner want to disallow a particular bot. This traffic-level view was essential when defining what should go into the managed robots.txt file.
Next steps for content owners
The managed robots.txt and the ads-based AI blocking are available to all Cloudflare customers, including free plan users. The company recommends adopting them now to take control over how website content is used. Looking ahead, Cloudflare is monitoring the IETF's pending proposal that would let publishers specify how automated systems may use their content, and plans to update its managed robots.txt accordingly. More granular AI bot controls and new distinguishing signals are also in the works as bot behavior gets harder to identify. Suspected abusive AI scraper activity can be reported through the submission form, contributing to shared knowledge about emerging threats.
