Why HTML is a poor fit for agent traffic
Traffic discovery is shifting away from classic search engines toward AI crawlers and agents, and those clients want structured data from a Web largely built for humans. The token economics explain the pressure: a markdown heading such as ## About Us costs roughly 3 tokens, while its HTML counterpart <h2 class="section-title" id="about">About Us</h2> burns 12-15, before counting the <div> wrappers, navigation bars and script tags that carry no semantic value on a real page.
Cloudflare measures the gap on its own material: a blog post that takes 16,180 tokens in HTML comes to 3,150 tokens in markdown, an 80% reduction. Markdown has become the common format for agents because its explicit structure suits AI processing and cuts token waste while improving results. But the Web is HTML, and page weight keeps rising, so agents must filter out non-essential elements to reach the relevant content. Converting HTML to markdown inside the pipeline wastes computation, adds cost and complexity, and may not match how the creator intended the content to be used.
Conversion at the edge via content negotiation
Cloudflare now performs real-time conversion at the source for enabled zones, using standard content negotiation. A client adds the Accept header with text/markdown among its options; Cloudflare detects the preference, fetches the original HTML from the origin, converts it, and returns markdown. This works from command-line clients and from agents built on Workers.
curl https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ \
-H "Accept: text/markdown"
const r = await fetch(
`https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/`,
{
headers: {
Accept: "text/markdown, text/html",
},
},
);
const tokenCount = r.headers.get("x-markdown-tokens");
const markdown = await r.text();
Coding agents including Claude Code and OpenCode already send these accept headers, so the response now simply arrives as markdown. Converted responses also carry an x-markdown-tokens header estimating the token count of the markdown document, which callers can use to size a context window or choose a chunking strategy.

Signals and defaults for converted responses
Markdown for Agents extends Cloudflare's Content Signals framework, announced during the last Birthday Week, which lets publishers state how their content may be used after access. Converted responses include the header Content-Signal: ai-train=yes, search=yes, ai-input=yes, indicating use for AI training, search results and AI input, including agentic use. Custom Content Signal policies are planned as a future option.
The feature is live on Cloudflare's Developer Documentation and Blog, so a request with Accept: text/markdown returns the converted document.
Alternatives outside the edge
For arbitrary document conversion, or when the content source does not run Markdown for Agents, Cloudflare offers two paths:
- Workers AI AI.toMarkdown(), which handles document types beyond HTML and also supports summarization.
- The Browser Rendering /markdown REST API, for rendering dynamic pages in a real browser before conversion.
Measuring markdown adoption in Radar
Cloudflare Radar now reports content type insights for AI bot and crawler traffic, both on the AI Insights page and on individual bot pages. A new content_type dimension and filter breaks down returned content types by MIME type category, and requests can be filtered per agent: the same view shows which requests return markdown to OAI-Searchbot, the crawler behind ChatGPT search. The data is available through Radar's public APIs and the Data Explorer.
Enabling the feature
To turn it on, open the Cloudflare dashboard, select the account and zone, find Quick Actions and toggle the Markdown for Agents button. The feature is in Beta at no cost for Pro, Business and Enterprise plans and for SSL for SaaS customers. Configuration details are in the Developer Docs.



