AGENTS.md outperforms skills in our agent evals

The setup: version-matched docs for Next.js 16

Coding agents are only as current as their training data. Next.js 16 introduces APIs like 'use cache', connection(), and forbidden() that models may not have seen. The result is incorrect code or fallbacks to older patterns — and the reverse problem when a project runs an older version and the model suggests APIs that don't exist yet.

We built an eval suite around Next.js 16 APIs and tested two ways to give agents version-accurate documentation:

• Skills: an open standard (agentskills.io) for packaging prompts, tools, and docs that an agent invokes on demand when it recognizes the need for framework-specific help.

• AGENTS.md: a root-level markdown file providing persistent context on every turn, so the agent never has to decide to load it. Claude Code has the equivalent CLAUDE.md.

We started with a bet on skills, then ran everything through the same eval suite.

Skills without a nudge performed like no docs at all

In 56% of eval cases, the agent simply never invoked the skill. The documentation was available; the agent chose not to fetch it. The skill-based configuration produced zero improvement over baseline — and on the Build/Lint/Test breakdown it actually scored worse on one metric (58% vs 63% on tests), suggesting an unused skill introduces noise rather than help.

This isn't a quirk of our setup. Agents not reliably using available tools is a known limitation of current models.

Explicit instructions helped, but the wording was fragile

Adding explicit instructions to AGENTS.md — telling the agent to use the skill — raised the trigger rate to 95%+ and the pass rate to 79%. But the exact phrasing mattered far more than expected.

Before writing code, first explore the project structure, 
then invoke the nextjs-doc skill for documentation.

Subtle wording changes produced dramatically different results. In one eval targeting the 'use cache' directive, the "invoke first" approach wrote the correct page.tsx but entirely missed the required next.config.ts changes; the "explore first" approach caught both. Small tweaks creating large behavioral swings felt too brittle for production use.

Hardening the eval suite

Our initial tests were unreliable: ambiguous prompts, assertions on implementation details instead of observable behavior, and coverage of APIs already present in training data. We removed test leakage, resolved contradictions, and switched to behavior-based assertions. The core of the suite is now Next.js 16 APIs that models haven't seen in training:

• connection() for dynamic rendering

• 'use cache' directive

• cacheLife() and cacheTag()

• forbidden() and unauthorized()

• proxy.ts for API proxying

• Async cookies() and headers()

• after(), updateTag(), refresh()

Every configuration was judged against the same tests, with retries to rule out model variance.

The passive context experiment

Instead of hoping an agent would invoke a skill, we removed the decision entirely. We embedded a compressed docs index directly in AGENTS.md — not the full documentation, just a structured pointer telling the agent where to find specific doc files matched to the project's Next.js version. A key instruction accompanied the injection:

IMPORTANT: Prefer retrieval-led reasoning over pre-training-led reasoning 
for any Next.js tasks.

The agent was told to consult these docs rather than rely on potentially stale training data.

Results: 100% with zero retrieval overhead

Across all four configurations in the hardened suite — baseline, unforced skills, forced skills, and the AGENTS.md docs index — the passive approach won outright:

The AGENTS.md configuration achieved perfect scores across Build, Lint, and Test. The "dumb" static file outperformed skill-based retrieval even when skill triggers were heavily tuned.

Our working theory for why passive context beats active retrieval:

1. No decision point. With AGENTS.md, there's never a moment where the agent asks itself whether to look something up — the information is already present.

2. Consistent availability. Skills load asynchronously and only on invocation. AGENTS.md content sits in the system prompt for every turn.

3. No ordering issues. Skills introduce sequencing decisions (read docs first vs. explore project first). Passive context sidesteps that entirely.

Compression handled the context concern

Bloat was the obvious objection, and it required work. The initial docs injection was roughly 40KB; compressing to 8KB (an 80% reduction) maintained the 100% pass rate. The compressed format is pipe-delimited, packing the full index into minimal space while covering every section of the Next.js documentation:

[Next.js Docs Index]|root: ./.next-docs
|IMPORTANT: Prefer retrieval-led reasoning over pre-training-led reasoning
|01-app/01-getting-started:{01-installation.mdx,02-project-structure.mdx,...}
|01-app/02-building-your-application/01-routing:{01-defining-routes.mdx,...}

The agent knows which files exist and where to find them, without the full content sitting in context. When specific information is needed, it reads the relevant file from the .next-docs/ directory.

Running it in your own project

Setup is a single command, now part of the official @next/codemod package:

npx @next/codemod@canary agents-md

That command detects your Next.js version, downloads matching documentation into .next-docs/, and injects the compressed index into your AGENTS.md. The approach works for any agent that respects AGENTS.md, not just Next.js-specific tooling.

Where skills still fit

Skills aren't useless. The AGENTS.md approach gives broad, horizontal improvements across all framework tasks. Skills are better suited to vertical, action-specific workflows that users explicitly trigger — version upgrades, App Router migrations, or applying framework best practices. The two approaches complement each other.

But for general framework knowledge, passive context currently outperforms on-demand retrieval. Practical recommendations from our evals:

• Don't wait for skills to improve. The gap may close as models get better at tool use, but results matter now.

• Compress aggressively. Full docs in context aren't necessary. An index pointing to retrievable files works just as well.

• Test with evals. Target APIs not present in training data — that's exactly where documentation access matters most.

• Design for retrieval. Structure docs so agents can find and read specific files without needing everything upfront.

The final goal is shifting agents from pre-training-led reasoning to retrieval-led reasoning. AGENTS.md turned out to be the most dependable mechanism we've found for that shift.