How we built AEO tracking for coding agents

Why AEO needs agent tracking

Standard chat models and coding agents consume web content very differently. For models, measuring visibility is comparatively simple: send prompts through AI Gateway to models like GPT, Gemini and Claude, then analyze answer quality, citations and search behavior.

Coding agents operate in a different context entirely. Rather than fielding a single API call, they run inside a project with filesystem access, a shell and package managers. Early sampling found that these agents perform web searches in roughly 20% of prompts, which makes source accuracy and response quality critical to track.

That introduces two main engineering challenges:

1. Execution isolation: Autonomous agents can execute arbitrary code, so each run needs a safe, contained environment.

2. Observability: Agents each have their own transcript format, tool naming conventions and output structure, making uniform capture difficult.

Running agents in sandboxes

Coding agents wrap their functionality in CLIs, which means they require a full runtime environment — not just an API endpoint. Vercel Sandbox provides ephemeral Linux MicroVMs that start in seconds, giving each agent run its own isolated environment.

Every agent follows the same six-step lifecycle:

1. Create the sandbox. Provision a fresh MicroVM with the appropriate runtime (Node 24, Python 3.13, etc.) and a hard timeout that kills hung or looping agents.

2. Install the agent CLI. Each agent ships as an npm package (for example, @anthropic-ai/claude-code, @openai/codex), installed globally as a shell command.

3. Inject credentials. Environment variables route all LLM calls through Vercel AI Gateway instead of giving agents direct provider API keys. This provides unified logging, rate limiting and cost tracking across every agent. Direct provider keys remain supported.

4. Run the agent with the prompt. Invocation patterns differ per CLI, but from the sandbox's perspective it is just a shell command.

5. Capture the transcript. Record what the agent did: tool calls, web searches and recommended responses. This step is agent-specific.

6. Tear down. Stop the sandbox even on failure via a catch block, preventing resource leaks.

import { Sandbox } from "@vercel/sandbox";
// Step 1: Create the sandbox
sandbox = await Sandbox.create({
  resources: { vcpus: 2 },
  timeout:  10 * 60 * 1000
});
// Step 2: Install the agent CLI
for (const setupCmd of agent.setupCommands) {
  await sandbox.runCommand("sh", ["-c", setupCmd]);
}
// Step 3: Inject AI Gateway credentials (via env vars in step 4)
// Step 4: Run the agent
const fullCommand = `AI_GATEWAY_API_KEY='${aiGatewayKey}' ${agent.command}`;
const result = await sandbox.runCommand("sh", ["-c", fullCommand]);
// Step 5: Capture transcript (agent-specific — see next section)
// Step 6: Tear down
await sandbox.stop();

Agents as configuration

Because the lifecycle is consistent, each agent is a declarative config object. Adding a new agent means adding an entry; the orchestration layer handles the rest.

export const AGENTS: Agent[] = [
  {
    id: "anthropic/claude-code",
    name: "Claude Code",
    setupCommands: ["npm install -g @anthropic-ai/claude-code"],
    buildCommand: (prompt) => `echo '${prompt}' | claude --print`,
  },
  {
    id: "openai/codex",
    name: "OpenAI Codex",
    setupCommands: ["npm install -g @openai/codex"],
    buildCommand: (prompt) => `codex exec -y -S '${prompt}'`,
  },
];

runtime selects the MicroVM base image, setupCommands runs prerequisite setup (for example, Codex needs a TOML config at ~/.codex/config.toml), and buildCommand returns the shell command for a given prompt.

Routing through the AI Gateway

The agents must not know they are being proxied. By overriding provider base URLs through environment variables inside the sandbox, all HTTP calls flow through AI Gateway to the actual provider.

For Claude Code, the override looks like this:

const claudeResult = await sandbox.runCommand(
  'claude',
  ['-p', '-m', options.model, '-y', options.prompt]
  {
    env: {
      ANTHROPIC_BASE_URL: AI_GATEWAY.baseUrl,
      ANTHROPIC_AUTH_TOKEN: options.apiKey,
      ANTHROPIC_API_KEY: '',  // intentionally blank as AI Gateway handles auth
    },
  }
);

ANTHROPIC_BASE_URL points to AI Gateway rather than api.anthropic.com. ANTHROPIC_API_KEY is intentionally empty — AI Gateway authenticates with its own token. The same pattern works for Codex via OPENAI_BASE_URL and other agents that respect base URL environment variables.

Normalizing transcripts

Raw transcripts describe everything an agent did, but no two agents record that information the same way. Claude Code writes JSONL files to disk, while Codex streams JSON to stdout and OpenCode uses stdout with yet another schema. Tool names, message nesting and response conventions all differ.

The normalization layer has four stages:

1. Transcript capture: Extract the raw transcript, which is agent-specific.

2. Parsing: Map tool names and message structures into a single unified event type.

3. Enrichment: Parse structured metadata — URLs, commands — from tool arguments, regardless of per-agent parameter names.

4. Summary and brand extraction: Aggregate events into statistics, then run the standard brand extraction pipeline.

Stage 1: Capturing transcripts

Claude Code persists its transcript as a JSONL file on the sandbox filesystem, so it must be located and read after the agent exits:

async function captureTranscript(sandbox) {
  const workdir = sandbox.getWorkingDirectory();
  const projectPath = workdir.replace(/\\//g, '-');
  const claudeProjectDir = `~/.claude/projects/${projectPath}`;
  // Find the most recent .jsonl file
  const findResult = await sandbox.runShell(
    `ls -t ${claudeProjectDir}/*.jsonl 2>/dev/null | head -1`
  );
  const transcriptPath = findResult.stdout.trim();
  return await sandbox.readFile(transcriptPath);
}

Codex and OpenCode both emit JSON to stdout, so capture involves filtering output for JSON lines:

function extractTranscriptFromOutput(output: string) {
  const lines = output.split('\\n').filter(line => {
    const trimmed = line.trim();
    return trimmed.startsWith('{') && trimmed.endsWith('}');
  });
  return lines.join('\\n');
}

Stage 1 outputs a raw JSONL string for every agent, but the internal structure still differs wildly.

Stage 2: Parsing

Each agent gets a dedicated parser that normalizes tool names and flattens message structures into a TranscriptEvent type.

Tool name variations are significant:

Parsers maintain lookup tables mapping agent-specific names to roughly 10 canonical names:

export type ToolName =
  | 'file_read' | 'file_write' | 'file_edit'
  | 'shell' | 'web_fetch' | 'web_search'
  | 'glob' | 'grep' | 'list_dir'
  | 'agent_task' | 'unknown';
const claudeToolMap = {
  Read: 'file_read', Write: 'file_write', Bash: 'shell',
  WebFetch: 'web_fetch', Glob: 'glob', Grep: 'grep', /* ... */
};
const codexToolMap = {
  read_file: 'file_read', write_file: 'file_write', shell: 'shell',
  patch_file: 'file_edit', /* ... */
};
const opencodeToolMap = {
  read: 'file_read', write: 'file_write', bash: 'shell',
  rg: 'grep', patch: 'file_edit', /* ... */
};

Event structures vary too:

• Claude Code nests messages inside a message property and mixes tool_use blocks into content arrays.

• Codex emits Responses API lifecycle events (thread.started, turn.completed, output_text.delta) alongside tool events.

• OpenCode bundles tool calls and results through part.tool and part.state.

After parsing, the output is a flat TranscriptEvent[] array with a consistent shape:

export interface TranscriptEvent {
  timestamp?: string;
  type: 'message' | 'tool_call' | 'tool_result' | 'thinking' | 'error';
  role?: 'user' | 'assistant' | 'system';
  content?: string;
  tool?: {
    name: ToolName;           // Canonical name
    originalName: string;     // Agent-specific name (for debugging)
    args?: Record<string, unknown>;
    result?: unknown;
  };
}

Stage 3: Enrichment and stage 4: analysis

Shared post-processing extracts structured metadata from tool arguments, so downstream code never needs to know whether Claude Code names a file path args.path or Codex calls it args.file:

if (['file_read', 'file_write', 'file_edit'].includes(event.tool.name)) {
  const path = extractFilePath(args);
  if (path) event.tool.args = { ...args, _extractedPath: path };
}
if (event.tool.name === 'web_fetch') {
  const url = extractUrl(args); 
  if (url) event.tool.args = { ...args, _extractedUrl: url };
}

The enriched events are aggregated into statistics (tool calls by type, web fetches, errors) and handed to the same brand extraction pipeline that processes standard model responses. From this point, the system cannot tell the data's origin.

Orchestration

The pipeline runs as a Vercel Workflow. Prompts tagged as "agents" fan out in parallel across all configured agents, each in its own sandbox:

export async function probeTopicWorkflow(topicId: string) {
  "use workflow";
  const agentPromises = AGENTS.map((agent, index) => {
    const command = agent.buildCommand(topicData.text);
    return queryAgentAndSave(topicData.text, run.id, {
      id: agent.id,
      name: agent.name,
      setupCommands: agent.setupCommands,
      command,
    }, index + 1, totalQueries);
  });
  const results = await Promise.all(agentPromises);
}

Findings so far

• Coding agents generate meaningful search traffic. Roughly 20% of sampled prompts triggered a web search, signaling that coding-agent optimization deserves attention.

• Recommendations differ in form. When an agent recommends a tool, it often generates working code — an import statement, config file or deployment script — rather than mentioning the tool in prose.

• Transcripts are unstable. Agent CLIs update rapidly, so an early normalization layer prevents constant breakage.

• Brand extraction is portable. Once normalized, agent transcripts and model responses flow through the same analysis pipeline. The difficulty is entirely upstream.

Next steps

• Open source the tool so teams can run AEO evals for both models and coding agents.

• Publish a methodology deep dive covering prompt design, dual-mode testing (web search vs. training data), query architecture and Share of Voice metrics.

• Expand agent coverage and prompt types — from tool recommendations to full project scaffolding and debugging workflows.