AI’s first year of real-world economics
When we launched Content Independence Day one year ago, the underlying tension was already visible. AI adoption was accelerating, referral traffic to publishers was falling, and crawlers were harvesting the web at scale—often without clear disclosure and almost never with compensation. We changed the defaults for new Cloudflare domains, blocking AI training crawlers unless site owners opted in. The goal was not to wall off the web but to create the conditions for a transparent, controlled, and ultimately compensated content economy.
That market has now arrived—and faster than most expected. The shift in how the Internet operates has been dramatic, and the data underscores just how quickly the old model has broken down.
Traffic has flipped to machines
Generative AI is being adopted at more than twice the speed of smartphones. Within 3.5 years, over 30% of humanity—2.5 billion active users—has integrated it into regular use.
The effect on human behavior online is severe. For every hour spent searching for information, only 15 minutes now occurs on the open web. Users increasingly go straight to AI for consolidated answers instead of visiting multiple sites. This year, agent traffic crossed a historic mark: more than 50% of all Internet traffic is now non-human.
Crawlers have a new job
Cloudflare’s crawler classification shows the change in purpose clearly:
- 52% of crawler requests are for AI training as of June 2026, up from 22% in Spring 2025.
- Mixed-use crawlers—combining search, agent use, and training—account for over 36% of activity.
- Pure search crawling is now a small and shrinking share, though it remains vital for publisher visibility.
This shift blurs the line between discovery and training. Content owners face an awkward choice: remain discoverable in the agentic era, or hand over their most valuable material without compensation.
The old bargain is broken
The open web’s historical model was simple. Creators gave search engines access to content in exchange for referral traffic, which they converted into revenue. That exchange is collapsing. Content is still being crawled, indexed, and consumed—but increasingly without sending users back to the source. When AI answers questions and completes research directly, publishers lose the audience that once sustained them.
This is not a media problem alone. Retail, software, IT, and finance are now affected; some of the most heavily crawled categories have seen human traffic drop as much as 40% in under a year.
Many publishers are preparing for what they call “Google Zero”—a future with little or no search referral traffic. Any organization publishing proprietary information online must now understand how to operate in this environment. The health of the Internet matters beyond individual businesses; it remains a critical public resource for surfacing information globally.
A market has formed
Our initial commitments were threefold: transparency and control for site owners, tools that create scarcity, and a marketplace where content and AI companies of all sizes can negotiate value efficiently. One year in, those pieces are assembling.
Cloudflare’s attribution, business intelligence, and enforcement tools gave publishers network-level visibility into AI access—far more effective than voluntary standards like robots.txt. For the first time, publishers could see how their content was being consumed and monetized. That control created scarcity, and scarcity produced leverage.
Publishers who restricted access gained negotiating power backed by operator-level evidence: how often LLMs attempted to crawl their content, which competitors were doing the crawling, their most in-demand URLs, and their crawl-to-referral ratios. This reduced the information asymmetry in licensing talks.
The results are measurable:
- More than 50 publisher-AI agreements have been signed since 2023.
- Major AI companies now actively license content.
- Collective licensing models are emerging and scaling.
- Large publishers are securing meaningful deals, confirming that content has real economic value in the AI ecosystem.
The debate has moved from whether content should be compensated to how.
The market is still immature
Licensing today remains largely bespoke and unlikely to fully replace lost referral, advertising, and affiliate revenue. Publishers are therefore optimizing for AI consumption alongside traditional discovery while exploring new monetization routes. Supply and demand are hard to match efficiently, and content valuation—everyone agrees not all content is equal—remains unresolved.
The Google convergence problem
Google still dominates discovery, accounting for roughly 88% of referral traffic. But it increasingly serves users content directly within its own AI experiences.
Discovery and consumption are fundamentally different. Search drives users to content; AI experiences summarize and reuse it without requiring a visit. Site owners treat these activities differently because one generates traffic and the other substitutes for it.
Most leading AI companies keep discovery crawlers separate from training crawlers, letting publishers enable access for one purpose without the other. Google does not. Its mixed-use crawler gives Google roughly 2x more access to information than leading AI companies, because publishers cannot participate in Google’s search ecosystem without also feeding its AI ecosystem. It also denies site owners the ability to see why Google is accessing their content, or to allow and block search and AI use independently at the network level. This lack of transparency is accelerating demand for new controls and monetization models.
A view from both sides
Cloudflare sits at the center of this shift. More than 20% of the web runs behind our network; 36% of the world’s most-visited sites and over 40% of the Fortune 500 are customers. Nearly 80% of leading AI companies use Cloudflare, as do thousands of developers and emerging firms. That vantage point lets us see both sides of the market—the content owners producing material and the AI companies consuming it—and trace the signals now connecting them.
Transparency is now a business requirement
Visibility and enforcement have moved beyond the security domain. Content owners need to know who is accessing their material, how it is used, and for what purpose — while AI companies increasingly understand that transparency reduces friction and builds trust with publishers. These factors now directly influence licensing negotiations and commercial decisions.
Cloudflare is continuing to invest in attribution, measurement, and publisher controls that give content owners more insight into content access and usage. The company argues that verifiable bot self-identification and declarations of crawl intent are foundational to a sustainable ecosystem. More than one-third of crawler activity on its network still comes from mixed-use bots that prevent content owners from distinguishing intent; Cloudflare is engaging with the ecosystem and investing in tooling with the goal of driving that number to zero within a year.
Better signals before better pricing
AI companies need more than content access — they need guidance on what to access, when, and how often content changes. Indiscriminate crawling wastes compute on the AI side and creates bandwidth burdens for publishers, reducing efficiency across the ecosystem. Cloudflare is investing in real-time freshness signals that carry trust, quality, and relevance data, helping AI companies find differentiated information while reducing unnecessary crawling.
Discovery must come before pricing. For the market to mature, publishers and AI companies each need better information about the other. Cloudflare is investing in market intelligence, content signaling, and discovery capabilities to lay the groundwork for more scalable market mechanisms.
Infrastructure for the agentic economy
A year after Content Independence Day introduced the idea that content owners should control how AI companies access and use their information, that control has produced a market. Transparency created scarcity; scarcity created leverage; leverage accelerated licensing. What was a theoretical discussion has become an active market with publishers, AI companies, and technology providers adapting to new economic realities.
The market is now entering a phase that demands new infrastructure. As the web becomes increasingly agentic, underlying systems must evolve to handle permissions, licensing, and commercial transactions at scale. Cloudflare believes these capabilities will converge into programmable, scalable mechanisms for content discovery and monetization — reducing friction while enabling richer value exchange. Its role is to build the infrastructure and business intelligence, and to contribute to the standards that allow the market to determine value more efficiently.
The data in this report is compiled from Cloudflare Radar and the Cloudflare Investor Day 2026 Presentation. Radar showcases global Internet traffic, attack, and technology trends, powered by data from Cloudflare's global network spanning 330+ cities in 100+ countries, plus aggregated and anonymized data from the company's 1.1.1.1 public DNS Resolver. More than 20% of the web sits behind Cloudflare's network.
