Why the web needs to learn a new language
The web has repeatedly evolved to serve new audiences: browsers, then search engines, and now AI agents. To help site owners keep up with this shift, we’re launching isitagentready.com, a tool that evaluates how well a site can be understood and used by AI agents. It covers everything from authentication flows and content access controls to content format negotiation and payment mechanisms.
We’re also adding a companion dataset to Cloudflare Radar that tracks how widely each agent-related standard is adopted across the Internet. This data refreshes weekly and is available through the Data Explorer and the Radar API.
Measuring agent readiness across the top 200,000 domains
To quantify the current state of agent-readiness, Cloudflare Radar selected the 200,000 most-visited domains, removed categories where the concept doesn’t apply (redirects, ad-servers, tunneling services), and ran the remaining sites through our new scanner. The results power the “Adoption of AI agent standards” chart now visible in the Radar AI Insights section.
The takeaway: the web is far from agent-ready. But this represents an opportunity — standards adoption is low enough that early movers can stand out.
Key findings from the initial scan
- robots.txt is nearly universal (78% of sites), but most instances remain written for traditional search crawlers rather than AI agents.
- Content Signals, a newer standard declaring AI usage preferences, is already present in the robots.txt of 4% of sites.
- Markdown content negotiation — serving
text/markdownwhen requested viaAcceptheaders — works on 3.9% of sites. - Protocols like MCP Server Cards and API Catalogs (RFC 9727) are found on fewer than 15 sites in the entire dataset.
The low adoption of emerging standards means there is plenty of room for sites to differentiate by acting early.
Setting an example: our own documentation overhaul
To show what agent-ready looks like in practice, we recently reworked Cloudflare’s Developer Documentation into one of the most agent-friendly sites available. The goal was not just compliance with standards, but tangible improvements for AI tools: quicker answers and significantly lower operational cost.
Scoring your site for agent readiness
Cloudflare has launched a public tool at isitagentready.com where you can enter any URL and receive an Agent Readiness score. The idea, as with Google Lighthouse for performance, is that a measurable score with actionable feedback accelerates adoption of new standards.
The scan sends requests to your site to check which agent-related standards it supports, then produces a score across four dimensions:
- Discoverability: robots.txt, sitemap.xml, and Link Headers (RFC 8288)
- Content: Markdown for Agents
- Bot Access Control: Content Signals, AI bot rules in robots.txt, and Web Bot Auth
- Capabilities: Agent Skills, API Catalog (RFC 9727), OAuth server discovery via RFC 8414 and RFC 9728, MCP Server Card, and WebMCP
Separately, the scan checks support for agentic commerce standards — x402, Universal Commerce Protocol, and Agentic Commerce Protocol — though these do not yet contribute to the score.
For every failed check, the tool outputs a ready-made prompt you can hand to your coding agent to implement the missing support.
The checker practices what it preaches. It exposes a stateless MCP server at https://isitagentready.com/.well-known/mcp.json with a scan_site tool over Streamable HTTP, letting any MCP-compatible agent run scans programmatically. It also publishes an Agent Skills index at https://isitagentready.com/.well-known/agent-skills/index.json with skill documents covering every standard it tests.
Discoverability
robots.txt has been the entry point for crawlers since 1994. For agents it serves double duty: it defines crawl rules and points to your sitemap XML files, which list every path on the site. The robots.txt file is the first place well-behaved agents look.
The Link response header (RFC 8288) is another discovery mechanism that works without parsing any page markup. Because the header ships with the HTTP response itself, agents can find linked resources such as sitemaps or API definitions directly:
HTTP/1.1 200 OK
Link: </.well-known/api-catalog>; rel="api-catalog"Content accessibility
Getting an agent to your site only matters if the agent can read your content. In September 2024, llms.txt was proposed as a plain-text file at the site root that functions as a structured reading list: what the site is, what is on it, and where the important content sits. It is essentially a sitemap written for an LLM to consume rather than a search crawler:
# My Site
> A developer platform for building on the edge.
## Documentation
- [Getting Started](https://example.com/docs/start.md)
- [API Reference](https://example.com/docs/api.md)
## Changelog
- [Release Notes](https://example.com/changelog.md)Markdown content negotiation goes further. When an agent requests a page with an Accept: text/markdown header, the server replies with clean markdown instead of HTML. That reduces token usage substantially — Cloudflare measured up to 80% token reduction in some cases — which makes responses faster, cheaper, and more likely to fit within the context windows most agent tools use by default.
The default scan only checks for Markdown content negotiation, not for llms.txt. You can opt into the llms.txt check when configuring a scan.
Bot access control
Once agents can navigate and read your content, the next decision is which bots you want to allow. The robots.txt file is the well-established place to declare crawl permissions down to specific paths. Content Signals lets you be more nuanced: a Content-Signal directive in robots.txt independently controls whether your content may be used for AI training (ai-train), for AI inference and grounding (ai-input), and whether it should appear in search results (search):
User-agent: *
Content-Signal: ai-train=no, search=yes, ai-input=yesFor the reverse case — letting friendly bots identify themselves — the Web Bot Auth IETF draft has bots sign their HTTP requests, with the receiving site verifying signatures against the bot's published public keys. Those keys are published at /.well-known/http-message-signatures-directory, which the readiness scan checks.
Web Bot Auth is only relevant for sites that run agents making requests to other sites; a site that merely serves content does not need it. But as more operators deploy their own agents, the standard is expected to grow in importance.
Protocol discovery
Beyond passive reading, agents can actively consume your APIs and execute tasks. The API Catalog (RFC 9727) creates a well-known endpoint at /.well-known/api-catalog where agents can discover all public APIs, with links to specs, docs, and status endpoints — no need to scrape a developer portal.
For tools, the Model Context Protocol is the open standard that connects AI models to external data and tools. Rather than building custom integrations per AI tool, you build one MCP server that any compatible agent can use. To help agents find that server, the MCP Server Card (currently a draft proposal) is a JSON file at /.well-known/mcp/server-card.json describing the server's tools, endpoint, and authentication requirements before an agent connects:
{
"$schema": "https://static.modelcontextprotocol.io/schemas/mcp-server-card/v1.json",
"version": "1.0",
"protocolVersion": "2025-06-18",
"serverInfo": {
"name": "search-mcp-server",
"title": "Search MCP Server",
"version": "1.0.0"
},
"description": "Search across all documentation and knowledge base articles",
"transport": {
"type": "streamable-http",
"endpoint": "/mcp"
},
"authentication": {
"required": false
},
"tools": [
{
"name": "search",
"title": "Search",
"description": "Search documentation by keyword or question",
"inputSchema": {
"type": "object",
"properties": {
"query": { "type": "string" }
},
"required": ["query"]
}
}
]
}Agents also perform better when they have Agent Skills tailored to a site's specific tasks. Cloudflare has proposed that sites publish available skills at .well-known/agent-skills/index.json, letting agents learn what is available and where to get it. Notably, the .well-known convention (RFC 8615) recurs across agent and authorization standards.
For sites behind authentication, the common workaround of handing an agent the user's logged-in browser session is risky. A safer pattern exists for sites that support OAuth: it can tell agents where to find the authorization server (RFC 9728), letting the agent send the human through a proper OAuth consent flow. Announced at Agents Week 2026, Cloudflare Access fully supports this OAuth flow, demonstrating with agents like OpenCode how the standard works when accessing protected URLs:
Commerce
Agents increasingly buy things on behalf of humans, but checkout flows were designed for human shoppers. x402 revives HTTP 402 Payment Required — a status code in the spec since 1997 — at the protocol level. The flow is straightforward: the agent requests a resource, the server returns 402 with a machine-readable payload containing the payment terms, the agent pays and retries. Cloudflare partnered with Coinbase to launch the x402 Foundation to shepherd x402 as an open payments standard.
The scan also looks for Universal Commerce Protocol and Agentic Commerce Protocol, two emerging standards aiming to let agents discover and purchase products the way humans do through ecommerce storefronts.
Cloudflare URL Scanner integration
Cloudflare's URL Scanner analyzes submitted URLs for HTTP headers, TLS certificates, DNS records, technologies, performance, and security signals. The same agent readiness checks now appear there as a dedicated Agent Readiness tab, showing which checks pass, the site's readiness level, and actionable guidance for improvement.
The integration is available via the URL Scanner API as well. Pass the agentReadiness option in a scan request to include these results programmatically:
curl -X POST https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/urlscanner/v2/scan \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
-d '{
"url": "https://www.example.com",
"options": {"agentReadiness": true}
}'Making Cloudflare’s own docs a model for agent-ready content
Having built the tools to score the Web at large, Cloudflare applied the same standards to its own documentation. The results are visible in its public Agent Readiness score, but the team went further, restructuring Developer Docs to be natively consumable by AI agents.
Markdown fallbacks via /index.md and rewrite rules
As of February 2026, only three of seven tested agents (Claude Code, OpenCode, and Cursor) send an Accept: text/markdown header by default. To cover the rest without duplicating static files, Cloudflare made every documentation page available as Markdown at a predictable URL: appending /index.md to any page path.
This is achieved dynamically with two Cloudflare Rules:
- A URL Rewrite Rule intercepts requests ending in
/index.mdand usesregex_replaceto strip that suffix, rewriting to the base path. - A Request Header Transform Rule inspects the original path (
raw.http.request.uri.path) before the rewrite and sets theAccept: text/markdownheader.
For any page, the Markdown source is therefore served unconditionally when requested via /index.md, regardless of the client’s headers. Example: https://developers.cloudflare.com/r2/get-started/index.md. These URLS are what Cloudflare points to in its llms.txt files, avoiding any additional build step or content duplication.
Hierarchical llms.txt that fits in context windows
A single llms.txt for a site with over 5,000 pages would exceed the context window of most LLMs. Instead, Cloudflare generates a top-level llms.txt that links to per-product files for each major directory (e.g., R2 and Workers).
The generation process also filters out low-value entries. Roughly 450 directory-listing pages, such as https://developers.cloudflare.com/workers/databases/, are excluded from the files.
Those pages appear in the sitemap and are crawlable but contain little more than redundant lists of links, since all child pages are already individually listed. Fetching them would only force an agent to make another request to find actual content.
Each entry that remains is built for machine consumption: a semantic name, a matching URL, and a weighted description drawn from existing frontmatter. No extra work is required for llms.txt generation—the metadata was already in the docs. The structure prioritizes context for the LLM over brevity for humans.
Testing against the afdocs spec
Cloudflare also validates its documentation against afdocs, an emerging open standard for agent-friendly documentation. Using that spec as a base, the team built custom audit tooling with minor patches specific to its site, resulting in a dashboard that surfaces content discovery and navigation issues.
Benchmarks: fewer tokens, faster answers
To measure the payoff, Cloudflare pointed an agent (Kimi-k2.5 via OpenCode) at the llms.txt files of several large technical documentation sites and posed specific technical questions. Against sites not optimized for agents, the agent working from Cloudflare docs consumed 31% fewer tokens and reached the correct answer 66% faster on average.
The difference comes down to structure. When product directories fit within a single context window, the agent can read the full index, pick the right page, and fetch it in a linear path. Sites without that structure force agents into an inefficient pattern:
- The grep loop: A single, oversized
llms.txtmay exceed the agent’s context. Unable to read the file in full, the agent searches it withgrep, and if the first query misses, it must reason and retry. - Narrowed context and lower accuracy: Iterative searching fragments the agent’s view of the documentation, reducing its overall understanding of the subject.
- Latency and token bloat: Each search cycle generates new “thinking tokens” and extra requests, slowing responses and inflating token usage—and therefore cost—for the end user.
Keeping AI crawlers off deprecated content
Historical pages like Wrangler v1 and Workers Sites remain online for humans, who can see prominent deprecation banners. But an LLM crawler might ingest that content without the visual cues and pass on outdated instructions.
Cloudflare’s Redirects for AI Training feature addresses this by identifying AI crawlers and redirecting them away from deprecated pages. The archive stays accessible to humans while agents only receive current implementation details.
Hidden directives and discoverable resources
Every HTML page in the docs carries a hidden HTML comment instructing LLMs to fetch the Markdown version instead of parsing the HTML. The directive points agents to the /index.md URL pattern, the Accept: text/markdown header approach, and the full index at llms.txt. It is deliberately stripped from the Markdown output to prevent a recursion loop.
For human developers, every product directory in the developer documentation now includes an “LLM Resources” entry in the sidebar, linking directly to its llms.txt, llms-full.txt, and relevant Cloudflare Skills.
Checking your own site
Agent readiness is becoming a baseline requirement for public technical content. Cloudflare’s own tooling is available at isitagentready.com, where site owners can obtain a readiness score and follow prompts to bring their content up to standard. Expect more data on agent standards adoption from Cloudflare Radar in the coming year.
