Why agents need simpler pages

When an agent fetches a webpage, it downloads the full browser experience: navigation, stylesheets, JavaScript bundles, tracking scripts, and footer links. All the agent actually needs is the page's structured text. The extra markup consumes its context window, confuses extraction, and raises the cost of every request.

Content negotiation solves this with a standard HTTP mechanism. The client declares its preferred format in the Accept header, and the server responds with the matching representation. Many agents already send Accept: text/markdown, so a server that supports content negotiation can return clean structured text from the same URL that serves HTML to a browser—no separate .md URLs or site-specific conventions required.

How the request flow works

Agents signal their preference by listing text/markdown first in the Accept header:

curl -H "Accept: text/markdown" https://example.com/page

When the header is present, the server routes the request to a markdown endpoint. When absent, the request falls through to the normal HTML page. Testing the behavior is straightforward with a command like the one above.

Implementation in Next.js

The setup requires two pieces: a rewrite rule in next.config.ts to inspect the header, and a route handler that serves the markdown.

The rewrite matches on the Accept header for every incoming request. If it contains text/markdown, the request is diverted to a dedicated markdown route instead of the default HTML page. The route handler then performs the conversion. For sites where content is authored in markdown, the handler can serve it directly without transformation. When content lives in a CMS as rich text, the handler converts to markdown on the fly.

The conversion preserves the meaningful structure: code blocks keep syntax highlighting markers, headings remain hierarchical, and links stay functional. Agents receive the same information as the HTML version, only in a format optimized for token efficiency.

Payload and performance impact

The comparison is stark: the HTML version of one page weighs around 500KB, while the markdown version is 3KB—a 99.37% reduction. Agents operating under token limits can consume substantially more content per request when the payload is that lean.

Keeping both versions in sync is handled via remote caching with shared slugs, so a CMS update refreshes the HTML and markdown representations simultaneously.

Markdown sitemaps for discoverability

Content negotiation extends to sitemaps. XML sitemaps are flat lists of URLs—no titles, no hierarchy, no indication of what a page covers. A markdown sitemap provides a structured table of contents with readable titles and parent-child relationships, letting agents understand the site's content and navigate to what they need.

The route handler for a blog sitemap generates a dated list of all posts. For documentation with nested sections, a recursive renderer preserves the hierarchy, making it clear which pages are children of which topics. Agents that don't send the Accept header still get a discovery path via a link rel="alternate" tag in the HTML <head>, pointing to the markdown sitemap.

Three paths to agent-friendly content

Content negotiation, markdown sitemaps, and alternate link tags give agents three independent ways to locate and consume a site's content efficiently. Any URL with support returns markdown when the requesting agent sends the proper Accept header, and the sitemap provides a directory for those who need to browse available resources first.