AI crawlers vs. the referral economy
Search engine crawlers were once a straightforward proposition for publishers: bots index content, users click through, and sites monetize visits. AI platforms follow a different model. Their bots crawl the same HTML pages, but they feed large language models (LLMs) that can repackage information for users without the user ever landing on the original site.
Cloudflare has been tracking this behavior, and has observed that visitor referrals from AI platforms are rare relative to the frequency with which those platforms scrape content. A new metric on the AI Insights page of Cloudflare Radar now quantifies that imbalance. The crawl-to-refer ratio lets site owners see, for each AI and search platform, how many HTML requests come from crawler user agents versus how many requests arrive from users who clicked a link on that platform's site or app.
How the ratio is computed
The ratio is built from two counts, both limited to responses with Content-type: text/html. The numerator is the total number of requests from user agents associated with a given platform. The denominator counts requests where the Referer header contained a hostname that Cloudflare attributes to that platform.
Platforms do not use a single user agent for all purposes. One agent might be used when content is fetched to train an LLM, while a different agent is used when content is fetched to satisfy a live user query, such as retrieving current flight information. Both agents are aggregated under the same platform name in this analysis.
On the referral side, a user who clicks a link inside an AI platform's web interface sends a request that includes the AI platform's hostname in the Referer: header. No such header is generated when a link is clicked inside a native mobile app, so referral counts undercount total clicks from AI applications. Web-based tools are the only source of referrals captured in this metric. Cloudflare believes this means the reported ratios may overstate the true crawl-to-refer balance, but not by a clear amount.
Observations from the first weeks of data
For the measurement period June 19–26, 2025, the metric shows wide variance across platforms. Purely by way of example, the ratio for Anthropic's Claude was roughly 70,900 crawls per single referral, whereas Mistral's ratio was 0.1:1 — that is, Mistral's referrals outnumbered its crawls by 10 to 1.
Ratios are not static, and week-over-week changes run in both directions. Compared with the prior week, DuckDuckGo and Yandex grew their ratios by more than 6%, whereas Google's ratio dropped 19.4%. The Google move appears tied to a drop in GoogleBot crawling starting June 24. Yandex's increase coincides with a rise in YandexBot activity observed since June 21.
Time series views of these ratios are available in Cloudflare Radar's Data Explorer, and the same data can be pulled from the Radar API.
Search still dominates referral traffic
Looking at the top of the funnel, referral traffic to sites is still dominated by search platforms. Data covering nearly the first four weeks of June 2025 shows Google referral share with a strong and consistent diurnal cycle. The google.* grouping combines referral traffic from google.com with data from country-specific domains such as google.es and google.com.tw.
Google referral traffic arriving from the company's own ASN (AS15169) is explicitly excluded from the analysis. That traffic is generated by prefetching under speculation rules, and does not represent a user actively choosing to open a page. In this dataset, that kind of traffic is not treated as genuine content consumption.
Other search platforms show similar diurnal patterns, though with referral shares that are a small fraction of Google's.
AI platform referrals, even combined, made up a much smaller share of referred traffic in June 2025 than any of the leading search platforms.
Shifts on the crawler side
Changes in ratio values over time often reflect bumps in crawling activity rather than shifting referral behavior. For much of June 2025, the share of crawl requests from Googlebot and GoogleOther declined, with several alternating peaks and valleys. A similar pattern appears in HTTP request traffic from Google's AS15169 over that same period.
OpenAI's GPTBot, for its part, showed multiple quiet stretches across the month during which little or no crawling activity was recorded.
What the numbers mean for publishers
The relationship between crawling and visitor delivery is now the central factor in the economics of web publishing. Search engines historically crawled a page far less often than they sent visitors to it; exposure to a crawler made a site's revenue model workable. When the ratio was fine, more crawling meant more clicks, and being available to the right bots was a net positive for a publisher.
That historical logic does not hold with AI platforms. The data points to AI models consuming content far more frequently than they send consumer traffic back toward the publishers who produced it.
Control mechanisms exist. In the past year, Cloudflare has released tools for blocking AI crawlers that train on site content, and more recently announced mechanisms intended to make the exchange between platforms and publishers more balanced. Site operators still need to decide on their own policy toward AI crawlers and then enforce it with audits and, if needed, blocks.
Versioned Bots directory expands
Separately, Cloudflare Radar has expanded its Verified Bots content into a standalone section. A new, dedicated Bots directory page shows each Verified Bot as a card listing its name, description, owning company, category and verification status. The directory can be searched by bot name, owner or description, and filtered by category — for example, limiting results only to bots used for monitoring and analytics.
Each card links to a bot-specific page which describes the user agent structure in HTTP request headers, recommended robots.txt directive syntax, and a graph of HTTP request volume over the chosen time span, with comparisons to the previous period included by default. Data is also retrievable from the Radar API, and changes to bot pages will documented on the Cloudflare Changelog.
