Cloudflare’s AI Index: Giving Site Owners Control Over AI Discovery

Cloudflare has announced a private beta for AI Index, a new service for domains on its network that acts as an AI-optimized search index. The goal is to shift the dynamics of content discovery: site owners get direct control over how AI systems access their data, and AI builders get a cleaner, permissioned alternative to blanket web crawling.

When enabled, AI Index automatically builds and maintains a searchable index of a domain. Cloudflare includes a set of standard tools and APIs with the index, and the site owner retains ownership and control over the index itself. Content can be monetized through Pay per crawl and the newer x402 integration. Internally, Cloudflare is also aggregating participating sites into a layer called Open Index, providing a single search point across multiple opted-in domains.

Why the Current Discovery Model Is Broken

The discovery process for AI largely depends on platform-controlled crawling. This creates two problems. For content creators, there is no efficient mechanism to signal when content is published or updated. For AI builders, crawling and re-crawling unstructured data is expensive and offers little insight into content quality or cost before retrieval begins. The search for discovery tools that bridge these gaps defines Cloudflare’s approach here.

Under the Hood of AI Index

The index is built on the same technology that powers Cloudflare AI Search (formerly AutoRAG). As pages are added or updated, they are processed in real-time. Cloudflare manages the infrastructure—compute, storage, embeddings, chunking, and models—so there are no individual components for the user to worry about.

Within the setup, control is a primary feature:

  • Site owners can include or exclude specific content from the index.
  • AI Crawl Control rules are enforced for access permissions.
  • Websites can opt out of AI Index entirely.

Once active, the index provides access through several standard interfaces:

  • An MCP server allowing agentic apps to connect via the Model Context Protocol, with support for Microsoft’s NLWeb tools.
  • A flexible search API that returns structured JSON results.
  • LLMs.txt and LLMs-full.txt files, offering a machine-readable map of the site according to emerging open standards.
  • A bulk data API for large content transfers in a single request, rather than per-document queries.
  • Pub-sub subscriptions for real-time content update events without requiring re-crawling.
  • Discoverability directives via robots.txt and well-known URIs so agents can find the available APIs automatically.

Monetization is integrated tightly with the index. Site owners can choose to charge for access using Pay per crawl or x402 interactions while monitoring who visits and manages permissions through the standard AI Crawl Control dashboard.

A Permissioned Web Feed for AI

The alternative Cloudflare is offering to builders is a pub/sub model. Instead of dispatching crawlers across the open internet, builders connect to specific websites that have opted in. The workflow for this new model is founded on a few steps:

  1. Discover opted-in websites by browsing a directory of sites making indexes available through Cloudflare.
  2. Evaluate content through metadata covering uniqueness, depth, contextual relevance, and popularity before fetching it.
  3. Pay fairly for access directly to the content creator via Pay per crawl.
  4. Subscribe to updates so new content references arrive without constant resource-draining re-crawls.

This approach is intended to reduce resource waste and provide higher-quality data while restoring control to content creators.

Scaling Up with the Open Index

Managing subscriptions to dozens of different sites can be a heavy lift for builders. The Open Index is an aggregate of participating individual site indexes, with features for filtering content based on quality, uniqueness, originality, and depth. For builders using the Open Index, the main benefits include:

  • Unified access: Query multiple participating sites simultaneously to reduce integration overhead, or use it as a ready-made web search layer.
  • Broader scopes: Use topic-specific bundles (news, documentation, scientific research) or a general discovery index across the larger web.
  • Bottom-up monetization: Results trace back to a specific site’s AI index, and payments flow to that respective site via Pay per crawl.

The combination of per-site indexes and the aggregated Open Index gives builders flexibility: granular control for training, agent data, or search experiences, plus broad coverage when a unified search across sites is required.

Cloudflare is opening the service to participants in a private beta. Sites can enroll to make their content available through AI Index. AI builders can also sign up to access the pub/sub web feed.