AI Crawlers Face a New Default: Pay Up or Get Blocked
Nearly three decades ago, Google’s founders built a search engine on a simple arrangement with content creators: allow your pages to be copied and indexed, and in return you receive traffic. That traffic could then be monetized through ads, subscriptions, or sheer readership. Google even supplied the tools to make that work—AdSense for advertising and Analytics for measuring the audience. For a long stretch, this bargain defined the web’s economy.
That contract is now dissolving. Google’s share of searches is falling for the first time in over a decade, and users are increasingly turning to AI-powered assistants for answers. The shift is visible within Google itself: AI Overviews and answer boxes now resolve many queries directly on the results page. Research indicates that about 75 percent of mobile searches already ended without a click to an external site, and AI-generated answers push that number higher.
The consequence is a dramatic erosion of traffic for publishers. Earning the same volume of visitors through Google today is roughly ten times harder than it was a decade ago. The situation is far worse with AI-native tools. Analysis from Cloudflare’s Radar shows that, compared with the old search model, securing a referral from OpenAI is about 750 times harder, and from Anthropic an astonishing 30,000 times harder. Users are increasingly consuming derivatives of original work rather than the originals themselves, and that dynamic undermines the value derived from ads, subscriptions, or audience growth.
Reversing the Default
The old exchange—copied content for traffic—no longer functions as a fair trade for most creators. Instead, AI crawlers strip-mine the web, extracting material while returning almost no value. To reset that relationship, Cloudflare is joining a coalition of major publishers and AI companies to flip the default behavior of its network: AI crawlers will be blocked unless they compensate creators for the content they ingest.
Rethinking Content Valuation
This is only the first step. The longer-term vision involves a marketplace where content creators and AI companies, regardless of size, can transact directly. The industry has long relied on traffic as a proxy for value, but that metric fails to capture what a piece of content actually contributes to an AI system.
Consider an AI model as a block of Swiss cheese. Its knowledge base has holes—gaps it cannot fill. Newly created content that fills one of those holes holds far more value than the repetitive, low-quality material that clutters most of the web today. That insight points toward a different scoring mechanism.
By evaluating content not on clicks but on its ability to advance knowledge—that is, by measuring how well it fills the gaps in an AI model’s understanding—the hope is to accelerate the improvement of AI systems while also incentivizing the production of high-value original work. The mechanics are not yet settled, and Cloudflare acknowledges it is consulting economists and computer scientists to work out the details.
A New Business Model for a Changing Web
The web is in transition, and so is its economic foundation. The opportunity is to salvage what worked over the past three decades and improve what did not. For Cloudflare, whose stated mission is helping build a better Internet, this initiative is part of that effort—ensuring that creators can demand fair value for the work they produce.
