The Internet has a second audience

A traffic curve that no longer tracks revenue

Cloudflare ended 2024 handling an average of 63 million HTTP requests per second. That figure has since nearly doubled to 115 million, with peaks beyond 150 million, and daily requests originating from AI agents on the network climbed more than 1,700% over the past year. For the first time, more than half of Internet traffic in a given year was not human.

Agents occupy a middle ground: not people, not the bulk automation of the past. They do not react to advertising, yet a person with a task usually sits behind them. Depending on how a site responds, they can be additive — or they can extract.

The arrangement that funded the web for thirty years — let search engines crawl, receive visitors, convert them — assumed discovery and payment were the same act. Answer engines break that assumption: they consume a page to produce a summary without sending a human to the place where ads or subscriptions live. In the most heavily crawled categories, including Retail, Computer Software, IT & Services, and Financial Services, human traffic has dropped by as much as 40% in under a year. Revenue per request falls while bandwidth, compute, and origin costs keep accruing, because a rising share of requests arrive with no referral, no ad impression, and no paying reader.

Blocking everything was the initial reflex — and in the previous year that meant recommending AI training crawlers be blocked on new domains. By stated purpose, AI training accounted for 22% of crawler requests observed in Spring 2025; by June 2026 that share had reached 52%. A blanket refusal, however, cannot carry the Internet economy now taking shape.

Tell the customer apart from the collector

What separates this traffic from ordinary crawlers is not volume. A training crawler arrives to build a model; an agent arrives because a specific person asked a specific question. Agent traffic inherits human patterns — a weekly rhythm, a summer lull — and its volume tracks how often people ask, not how often a publisher writes.

None of the machinery needed to serve such visitors existed before: identifying them, distinguishing them from one another, imposing terms, charging them. Four capabilities, one longstanding gap.

Identification

Grouping traffic under the label "AI bot" conveys nothing; behavior is the meaningful signal. AI Crawl Control, Business Insights, and BotBase surface which crawlers arrive, what they take, what returns, and which URLs they target most. Names alone are worthless unless verifiable, so Web Bot Auth lets operators — OpenAI, Google, and AWS among them — cryptographically sign requests, separating genuine agents from impersonators without relying on IP ranges or user-agent strings. More than 500 billion verified bot requests are seen each week.

Terms

A single "block AI bots" toggle was replaced in July by distinct Search, Agent, and Training controls, offered on every plan including Free. Usage data explained the need: under 1% of sites block search crawlers, while 17% block training. Publishers were not trying to be invisible; they wanted visibility without exploitation, and as agentic traffic and new uses of information grew, they had neither transparency nor choice.

Mixed-use crawlers complicate matters — a bot that performs both search and training cannot be refused on one count alone. Disallow AI Training, shipped September 15, preserves search indexing while instructing the operator through crawler-specific mechanisms not to train on the content; Apple, Google, and Microsoft have committed to honor it. Cloudflare Radar tracks crawler behavior publicly. Recommended configurations on new domains now follow how a site earns money rather than which software is visiting — for ad-supported sites, disallowing training and blocking agents on ad-bearing pages, since an ad pays only when a human sees it. Settings remain changeable at any time.

Payment

The Agentic Internet described in August 2026 was characterized as readable, discoverable, callable, and payable. Payable is the property that decides whether the open web can fund itself, since it requires a way to answer "yes, if you pay" rather than a binary yes or no. The licensing market demonstrates both appetite and shortfall: more than 50 publisher-AI deals signed since 2023, nearly all bespoke and bilateral, between large publishers and large AI companies. Value is proven; reach is not. Most of the web, and most buyers, sit outside those agreements.

Different assets warrant different models. High-value content and datasets call for a trusted network in which buyers are identified and report usage. APIs and MCP tools work differently: the request is the use. Both are being built.

Pay Per Use reaches sites no direct deal will ever cover — no AI company can negotiate with millions of publishers, and direct licensing cannot scale to them. Rather than charging for the crawl, it pays when content is used. Verified-crawler buyers define what counts as use and what they will pay; publishers review the offer, opt in or out at will, and can withdraw if it stops working. The buyer reports each use, Cloudflare verifies the reports, bills the buyer, and pays the publisher. Reporting carries as much weight as money: publishers learn what was used, when, what they earned, and — where the buyer discloses it — which questions surfaced their work, a feedback loop that shapes coverage decisions. Definitions of use will vary by context: a search engine citing a source, a research agent quoting a passage, and a shopping agent completing a purchase create different value and will want different business models. Multiple models can run over the same rails without publishers integrating anything new, so a trade journal for marine engineers with a few thousand subscribers and no licensing prospects still gets paid by every participating AI company that draws on its work.

Monetization Gateway, in closed beta for eligible U.S. Cloudflare customers, prices anything passing through the network in the Rules language customers already know. On a rule match, it returns HTTP 402 Payment Required over the open x402 protocol, and the agent pays the seller directly. This addresses a class of value that never had a transfer mechanism: accounts, API keys, and subscriptions assume a known customer, not an unfamiliar agent wanting a single lookup. Pricing can be per request, per query, or per token, at fixed or capped rates — a sports statistics site built on ads, for example, charging a fraction of a cent whenever an agent asks who leads the league in assists. Thousands of sellers joined the waitlist after the announcement, most requesting that agents be charged rather than humans. Cloudflare's own AI Gateway uses Monetization Gateway so agents can pay for inference, exposing rough edges before customers encounter them.

For buyers, both beat a block page: dependable access and a route to millions of sites without a deal or API key apiece, with a receipt for every paid request. Both are bets, built on shared primitives — identity, metering, pricing, settlement, analytics — and both work together, allowing a publisher to disallow training, allow search, earn from AI answers, and charge per article from one dashboard. Because pricing and discovery remain unsolved, both launch as betas.

Efficiency

Falling revenue and rising cost are separate problems, and much of the cost is waste: crawlers repeatedly downloading human-oriented pages to extract a few paragraphs, often re-crawling unchanged sites, burning site bandwidth and crawler compute before any answer is produced. Bandwidth consumed per operator is now visible in the dashboard. A joint research project with OpenAI announced in July is a first-of-its-kind pilot examining how network-wide insights can help AI search engines discover and index open web content more efficiently; initial results are expected within weeks. For customers, Markdown for Agents strips human-oriented styling so agents can read a page, and WebMCP lets a site expose actions directly instead of forcing agents to guess at buttons.

Open rails, or a rented one

Over 20% of the web and nearly 80% of leading AI companies sit behind the same network, which gives a view of both sides of the market: rails for visibility, identity, controls, and settlement, with the market left to determine worth. The old bargain has ended; the replacement is unwritten. Two futures are available. In one, a handful of companies decide how agents find things, prove identity, and pay, and everyone else routes through them. In the other, those functions are open standards any site can implement, and a site of any size can set terms and get paid. The rails here are built on open standards such as x402 and Web Bot Auth so others can build on them, with domain owners choosing their own identity providers, payment processors, and agent partners — Cloudflare as one option rather than the entire stack.