The search bargain is broken
For nearly three decades, the web ran on a straightforward trade: search engines crawl and index your content, and in return you get visitors you can monetize through ads, subscriptions, or audience building. Discoverability and revenue were two sides of the same coin. The rise of AI answer engines has broken that exchange. These systems read a page and deliver a synthesized summary directly to the user, eliminating the need for the visit — and the business model that depended on it.
The evidence is stark. A 2025 Pew Research Center study found that when Google displays an AI summary, users click a traditional search result link just 8% of the time — roughly half the rate without a summary — and click links inside the summary only 1% of the time. This leaves publishers with an impossible choice: block AI systems and become invisible, or allow access and give away their content's value for little in return. More than 50% of online traffic is now non-human, and the challenge is no longer a handful of identifiable training crawlers; search itself is being rebuilt around AI answers.
From defense to offense
Cloudflare has spent the past year giving website owners tools to control bot access and enforce standards through its responsible AI bot principles. These measures say "no" effectively, but they don't rebuild the revenue models that sustain content creation. Two new initiatives Announced today aim to shift from protecting content to monetizing it properly.
Making AI search smarter
A new research program will use signals gathered across Cloudflare's network — which sits in front of more than 20% of the web — to help answer engines find fresher, more relevant content while reducing wasteful crawling. The program combines signals customers choose to share about content freshness with Cloudflare's own visibility into human and bot traffic patterns.
Cloudflare data indicates that over 50% of crawl traffic from well-behaved bots goes to re-fetching pages that haven't changed, a figure likely to grow as crawl volumes increase. A simple freshness signal can let crawlers skip those pointless trips, cutting compute costs for AI companies and reducing server load and bandwidth expenses for site owners. The program aims to measure two outcomes: whether these signals help answer engines surface better content, and how much unnecessary crawling they eliminate.
The initiative is neutral by design, intended to work with any answer engine that follows fair play rules. It is limited to search — no content is shared, and nothing is used to train foundation models. Cloudflare plans to publish findings and make the capability broadly available later this year.
From Pay Per Crawl to Pay Per Use
Last year's Pay Per Crawl model let publishers charge AI companies for bot access, but crawling is a poor proxy for value. A page might be fetched once and cited in thousands of answers, or crawled repeatedly yet never used. To address this, Cloudflare is evolving the model into Pay Per Use, running experiments with AI companies including Ceramic.ai and You.com.
Ceramic.ai has built a "pay-per-query" model where publishers who opt in get paid when their content appears in search results. This ties compensation to the value the work delivers rather than the frequency of crawler fetches. Participating content owners also gain access to new reporting dashboards for answer engine optimization (AEO), showing top queries, specific pages and snippets that appear in results, average ranking positions, and more. You.com is testing a different approach, allowing agents to pay on demand for specific premium content without upfront commitments.
These are experiments, and there's much to learn about how such models hold up at internet scale. But the direction is clear: answer engines get fresher, better-grounded results, and content creators get paid when their work contributes to those results. Cloudflare's role is to provide the infrastructure layer that makes this market possible.
The old model optimized for human search — excerpts, ten blue links, and a click. Agentic search is different: an agent can read fast and search continuously, making dozens of queries to answer one question. Search becomes a utility rather than a destination. In that world, the meaningful unit isn't the crawl or the click — it's the outcome. Pricing the outcome and paying the people who made it possible is how the web continues to be worth contributing to.
