The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals

AI Training Crawls Grow, but Referral Traffic Shrinks

Generative AI has fundamentally changed the relationship between content creators and the bots that index their sites. Search engines once rewarded publishers with clicks; today, AI training crawlers consume vast amounts of data without sending many users back. Data from Cloudflare’s network shows the gap is widening: while crawling activity by search engines and AI services surged in early 2025, referral traffic from Google to news sites has fallen since February of that year.

Key Findings at a Glance

• Training dominates: Training now accounts for nearly 80% of all AI bot activity, up from 72% a year ago.
• Referrals decline: Google referrals to news sites dropped roughly 9% in March 2025 compared with January, and 15% in April.
• Growth cools: AI and search crawling rose 32% year-over-year in April, but slowed to just 4% by July.
• Bot share shifts: OpenAI’s GPTBot more than doubled its share of AI crawling traffic (from 4.7% to 11.7%); Anthropic’s ClaudeBot grew from 6% to roughly 10%, while ByteDance’s Bytespider fell from 14.1% to 2.4%.
• Crawl-to-refer imbalance persists: Anthropic leads in crawls per referral, with 38,000 page crawls per visitor in July (down from an extreme 286,000:1 in January). Perplexity’s referrals decreased even as its crawling increased.

Google Referral Slump Coincides with AI Overviews

Referral traffic from Google to news sites in Cloudflare’s cohort — spanning the Americas, Europe, and Asia — has seen a clear decline since February 2025. This downturn deviates from typical seasonal patterns, where overall traffic and referrals drop only during the summer months of July and August. The decline was sharpest in March, a 31-day month that delivered almost the same referral volume as the shorter, 28-day February.

This period aligns with a series of Google changes. AI Overviews first launched in the U.S. in May 2024, and in March 2025, Google upgraded the feature with Gemini 2.0, introduced AI Mode in Labs, and expanded Overviews to more countries in Europe. By May 2025, AI Mode had rolled out broadly in the U.S. with Gemini 2.5, adding conversational search and personalized recommendations. The timing suggests the search-to-news pipeline is being weakened by AI-driven results.

Crawl Growth Spikes in Spring, Then Levels Off

Using a fixed set of Cloudflare customers to remove bias from network growth, the data shows two distinct spikes in AI and search crawling: one in November 2024, and a second between March and April 2025. April alone was up 32% compared with May 2024, the first full month with comparable data. After that surge, activity cooled — June remained 24% higher year-over-year, but July’s growth had fallen to just 4%.

The pattern aligns with general internet seasonality, where crawler activity rises in November with shopping and seasonal habits, and quiets during the Northern Hemisphere’s summer. The 2024 November uptick repeated the pattern seen earlier for AI and search bots.

Googlebot Holds, But AI Crawlers Gain Ground

Googlebot is still the anchor of all AI and search crawler traffic, holding a 39% share in July 2025. However, the fastest-growing bots are AI-specific. GPTBot’s share grew from 4.7% in July 2024 to 11.7% a year later. ClaudeBot increased from 6% to nearly 10%, and Meta’s crawler jumped from 0.9% to 7.5%. In contrast, Amazonbot dropped from 10.2% to 5.9%, and ByteDance’s Bytespider fell from 14.1% to just 2.4%.

AI-Only Bots: GPTBot Surges Past the Competition

Restricting the view to AI bot traffic alone makes the shift more pronounced. Since January 2025, GPTBot has steadily increased its crawling volume, driven mainly by training activity. ClaudeBot’s crawling accelerated in June, while Amazonbot and Bytespider both slowed. A comparison of July 2024 and July 2025 shows GPTBot gaining 16 percentage points, Meta’s crawler rising by more than 15, and ClaudeBot growing by 8. Amazonbot dropped 12 percentage points, and Bytespider lost over 31 percentage points.

Training Dominates Crawl Purpose

Cloudflare classifies crawler purpose based on operator disclosures and industry sources. Over the past 12 months, training accounted for 80% of all AI crawling, compared with 18% for search and just 2% for user actions. In the last six months, training’s share rose further to 82%, while search fell to 15% and user actions increased slightly to 3%.

The year-over-year comparison underscores the trend. In July 2024, training drove 72% of AI crawling; by July 2025, it reached 79%. Search fell from 26% to 17% over the same period, while user actions grew modestly from 2% to 3.2%.

For publishers, the takeaway is clear: the bots that crawl the most pages are rarely the ones delivering readers. With training-related crawls growing and search referrals diminishing, the economics of web content are shifting — and not in favor of those who create it.

Why a growing crawl-to-refer gap signals trouble for the open web

The crawl-to-refer ratio — pages crawled for every human visit a platform sends back — exposes a central tension in the AI content economy. A high number means a company consumes vast amounts of content while returning very little traffic. For publishers, the calculation is stark: feed the machine and get nothing measurable in return.

Cloudflare’s data from January through July 2025 shows Anthropic as the most extreme case. Even after an 87% decline over the period, its crawlers still visited roughly 38,000 pages for every referral in July — the widest imbalance among major AI operators. The improvement tracks with Anthropic launching web search in Claude in March 2025 (initially for U.S. paid users, later expanded globally by May), which introduced clickable citations and opened new referral paths.

The full-period snapshot reveals divergent trajectories:

• Anthropic saw the steepest drop in bot-to-human traffic, falling 86.7% from 286,930 bots per human in January to 38,065 in July. The ratio remains by far the highest, with tens of thousands of pages crawled per referral.
• Perplexity moved in the opposite direction, with crawling up +256.7% relative to human visitors — from 54 bots per human in January to 195 in July. Still far below Anthropic, the trend signals increasingly aggressive extraction relative to the traffic returned.
• OpenAI improved modestly, from 1,217 to 1,091 bots per human (-10%). The change suggests slightly more referral traffic relative to crawling activity.
• Microsoft remained essentially flat, moving from 38.5 to 40.7 bots per human (+6%), indicating stable behavior from its Bing-linked services.
• Yandex climbed from 15.5 to 21.4 bots per human (+38%) — still a small ratio overall, but showing increasingly heavy crawling relative to referrals.

Verification remains an open question

Beyond volumes, Cloudflare’s data highlights a compliance gap. Most leading AI crawlers appear on its verified bots list — their IP ranges match published data and they respect robots.txt. But adoption of newer standards such as WebBotAuth, which cryptographically signs HTTP messages to confirm a request comes from a specific bot, is essentially absent. Meta, OpenAI, and Anthropic run separate bots for different functions; Google and Microsoft rely on unified crawlers. Anthropic in particular lags on verification, leaving its crawler easier to spoof and its actual compliance unclear — a problem when bad actors can masquerade as legitimate bots and ignore robots.txt entirely.

If training-driven crawling continues to outpace referrals, publishers face a grim arithmetic: their content powers AI answers while yielding nothing back in audience or revenue. The open web is at a decision point. A new equilibrium could emerge — one where AI systems help sustain the creators they depend on — or the web could become a one-way training set, drained of value with little returning to those who produce it. Cloudflare's updated AI Insights page on Radar offers a live window into which direction the industry is taking.