Beyond the Binary: Paid Access for AI Crawlers
Publishers and website owners face a frustrating dilemma: either let AI systems consume their content for free or block all automated access entirely. There's no middle ground for those who want their work used, but want compensation for it. Currently, that requires negotiating individual deals, which is only feasible for organizations with significant scale and leverage.
Cloudflare's answer is to revive a mostly unused piece of web infrastructure: HTTP response code 402 Payment Required. Their new pay per crawl system, now in private beta, uses that status code to create a programmatic framework for charging AI crawlers per request.
How Pay Per Crawl Works
Pay per crawl builds on existing web infrastructure. Cloudflare is the Merchant of Record for transactions and provides the underlying technical system. Each time an AI crawler requests content, it either presents payment intent via request headers and receives a successful HTTP 200 OK response, or it gets a 402 Payment Required response with pricing information.
Domain owners configure a flat, per-request price for their entire site. For any given crawler, publishers have three options:
- Allow: Let the crawler access content free of charge.
- Charge: Require payment at the configured price.
- Block: Deny access completely, without an option to pay.
If a crawler has no billing relationship with Cloudflare and therefore can't be charged, a publisher can still set it to "charge." That functions as a network-level block — an HTTP 403 Forbidden where no content is returned — but informs the crawler that a paid relationship may be possible later.
A key flexibility: publishers can exempt specific crawlers from charges. This is useful for letting certain crawlers through for free or when a content partnership is negotiated outside the pay per crawl system.
Cloudflare enforces Allow or Charge rules only after existing WAF policies and bot management or bot blocking features have already been applied. This ensures integrations work with a publisher's existing security posture.
Crawler Identity and Authentication
A central technical challenge was proving a request truly came from a specific crawler, preventing spoofing. The solution uses the Web Bot Auth proposals. Crawlers must:
- Generate an Ed25519 key pair and make the JWK-formatted public key available in a hosted directory
- Register with Cloudflare, providing the key directory URL and user agent information
- Configure the crawler to use HTTP Message Signatures with each request
Once registration is accepted, requests must include signature-agent, signature-input, and signature headers. This identifies the crawler and allows it to discover paid resources.
GET /example.html
Signature-Agent: "https://signature-agent.example.com"
Signature-Input: sig2=("@authority" "signature-agent")
;created=1735689600
;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"
;alg="ed25519"
;expires=1735693200
;nonce="e8N7S2MFd/qrd6T2R3tdfAuuANngKI7LFtKYI/vowzk4lAZYadIX6wW25MwG7DCT9RUKAJ0qVkU0mEeLElW1qg=="
;tag="web-bot-auth"
Signature: sig2=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQdi5N5YYKrD+mCT1HA1nZDsi6nJKuHxUi/5Syp3rLWBA==:Two Payment Flows
Whether content requires payment can be determined through two approaches.
Reactive: Discovery First
If a crawler requests a paid URL, Cloudflare returns an HTTP 402 Payment Required response along with a crawler-price header. The crawler can then decide whether to retry, this time including a crawler-exact-price header to indicate its agreement to pay the stated price.
HTTP 402 Payment Required
crawler-price: USD XX.XXProactive: Intent First
Alternatively, a crawler can declare payment intent upfront by including a crawler-max-price header on its initial request. If the price configured for a resource is at or below that limit, the request goes through and content is served with a successful HTTP 200 OK, confirming the charge.
GET /example.html
crawler-max-price: USD XX.XXThere's a nuance to the pricing mechanics. If a crawler's crawler-max-price exceeds the site owner's configured price, only the configured price is charged. But if the configured price is above the crawler's maximum, the response is 402 Payment Required specifying the actual cost.
Only one declaration header is allowed per request — either crawler-exact-price or crawler-max-price. If the request is valid and all checks pass, content is served and the crawl event is logged. Any invalid aspect returns a 402 response.
Settlement and Billing
Both crawler operators and content owners must save payment details in their Cloudflare accounts. Billing events are logged each time a crawler makes an authenticated request with payment intent and receives an HTTP 200-level response containing a crawler-charged header. Cloudflare aggregates the events, charges the crawler, and distributes the earnings to the publisher.
GET /example.html
crawler-exact-price: USD XX.XX From Crawlers to Agents
Pay per crawl begins with crawlers, but the architecture is aimed at an agentic future. The choice of HTTP 402 was deliberate: it allows for programmatic negotiation of access to digital resources.
Cloudflare expects the system to evolve considerably. Potential directions include path-specific or content-type-based pricing, dynamic pricing based on demand or on how many users an AI application has, and granular licensing at internet scale for training, inference, search, or other use cases entirely. In an agentic context, a user could give a research agent a budget to pay for access to the best content on a topic, whether that's the latest cancer research, legal materials, or restaurant recommendations.
Pay per crawl is currently in private beta. Interested crawler operators or content owners can sign up at http://www.cloudflare.com/paypercrawl-signup/; existing Enterprise customers can contact their account executive.
