Token counts are the wrong yardstick for agent efficiency

When working with AI coding agents, token count per interaction is not a meaningful efficiency measure. The real goal is moving a task forward with the right context — not reducing tokens at any cost. Concise output that strips needed information forces the agent to make extra calls or rerun commands, ultimately making the task slower and more expensive.

GitHub Copilot shipped four changes reflecting this principle: preserving useful context while cutting repetitive output, removing unused formatting, compressing prompts without losing behavior, and delivering completed background work without requiring an extra retrieval step. Each change was first evaluated offline using agentic coding benchmarks, then validated through controlled online experiments. Examples below come from GitHub Copilot CLI, but the underlying harness is shared by the Copilot app and Copilot code review.

Chart showing 3.1% 'Remove view previxes', 5.5% 'Selective output compaction', 2.9% 'Compact task-tool prompt', and 2.3% 'Reduce notification roundtrips'.
Figure 1: Four independent A/B experiments using the same AI-credit metric. The segments are shown together for comparison; their effects are not necessarily strictly additive. 

Why short tool output can cost more globally

Utilities such as RTK (Rust Token Killer) shorten shell output before an agent reads it. Tested with the Copilot agentic benchmark configuration, RTK shortened some responses, but when omitted text mattered, the model sometimes reopened the original output or reran the command to recover what it needed. Those recovery steps added turns and carried more context forward. Individual tool responses were shorter, but tasks on average used more tokens and took longer. Tokens were saved locally and spent globally.

Flow chart showing: RTK, compresses shell output > Local win, tool output gets shorter > Useful detail is missing > Recovery, reread or rerun > More turns and context carried forward. Then the option of finishing at 'End-to-end result, Tokens and cost up, Task duration up, Task completion: steady,' or 'Recovery repeats' going back to 'useful detail is missing'.
Figure 2: A shorter tool response can make the completed task more expensive when missing details force the agent to reread output, rerun commands, and carry more context forward. 

This result is specific to the integration and workloads tested, not a blanket judgment on RTK or output compression. The lesson is that tokens per tool call is the wrong objective; efficiency changes must be evaluated end-to-end, from user request to final result. The useful question is what can be removed without making the model repeat work.

Selective compression: remove noise, keep information

Analysis of benchmark runs showed that install, build, test, and lint output often contains repetitive noise, while source-like output and arbitrary command results are more likely to hold information an agent needs. That informed a selective output compressor, in part inspired by RTK and similar tools. It was evaluated on agentic benchmarks and open source repositories exercising their build, test, and lint systems.

Early versions were too aggressive and made the model repeat work or read full saved output, raising end-to-end cost and hurting task success. Compressing git diff was removed from the filter set after agents reopened the original output to recover missing information. Iteration led to a three-part policy:

  1. Preserve source-like and arbitrary output. Commands such as cat, git diff, git show, and arbitrary scripts come back unchanged.
  2. Reorganize search results without dropping content. Matches and file lists from grep and similar tools are grouped more efficiently while retaining every result.
  3. Compress repetitive noise selectively. Install, build, test, and progress output is compressed only when savings are substantial.

The shipped version is conservative — not because conservatism was a goal, but because evaluations supported it. When output is compressed, the agent retains a direct recovery path to the complete original.

Flowchart showing how GitHub Copilot handles shell-command output. Copilot calls a shell command, classifies the output, then chooses one of three paths: keep arbitrary/source output unchanged, reorganize search results without losing any matches, or selectively compress repetitive noise (like install/build/test logs) while preserving full output and providing a recovery path. The processed result is returned to Copilot.
Figure 3: The shipped compressor preserves source-like output, reorganizes search results without loss, and compresses only predictable repetitive noise while retaining the full original.

Recovery rate was tracked as an evaluation signal: if the agent opened saved originals, reran commands, repeated exploration, or took extra turns, the compressor likely removed something valuable. On offline tasks where compression triggered, no statistically significant task-success regression was detected and agents very rarely opened saved originals. In the online experiment, average cost decreased slightly with no material quality regression.

Line numbers removed from a tool that didn't use them

The view tool, used by agents to read files into context, previously prefixed each line with a number. Earlier file-editing tools relied on those numbers to target changes, but current tools match on surrounding code instead. The prefixes persisted even though the normal workflow no longer used them. Small per-line, the wasted formatting accumulated across every file read in a session.

Before-and-after image of code snippets. The line-number prefixes re removed from the 'After' image.
Figure 4: Removing line-number prefixes preserves the source exactly while eliminating formatting that was repeated across every file read.

Line numbers remain useful in diffs and short snippets, but attached to every file read they served no current purpose. Removing them cut model-inference cost roughly 5% in offline agentic benchmarks, with success rates staying within run-to-run variance and no increase in edit failures. A Copilot CLI online experiment confirmed the results: about 3% lower average daily model-inference cost per user, with no material quality or satisfaction regression. Developers gain more context window for real work instead of unused formatting — no new instructions, no information to recover, no decisions for the model.

Shorter prompts only work if behavior survives

Prompts shape agent behavior and go to the model every turn. Compression only pays off if the agent keeps essential behaviors. In GitHub Copilot, the task tool's guidance had accumulated across tool descriptions, schemas, agent definitions, system instructions, and companion tools. A meta-prompting loop, iterating on its own prompt with targeted behavioral tests, reduced it by roughly half.

The first online experiment caught a regression initial offline evaluations missed: the loop rewrote cautious parallelism guidance into a hard scheduling policy, causing independent custom agents to run sequentially. The experiment was stopped and a regression evaluation written for the exposed behavior. The fix replaced an explicit allowlist and denylist with one sentence:

Independent agents can run in parallel; consider side effects.

That one line was shorter and less restrictive, deferring the parallelism decision to the model. The behavior test passed, existing behavioral tests stayed green, and the shipped prompt removes about 1,300 task-tool prompt tokens per turn — about 1.8% fewer total prompt tokens per session and 2.9% lower normalized cost per active hour, with no quality regression.

Three-stage diagram labeled Compression → Regression + fix → Completed. Left panel shows an original prompt compressed by about 50%. Middle panel highlights a regression where agents became serialized, then a fix by editing one sentence to restore parallelism. Right panel shows final shipped prompt with restored behavior and cumulative savings of about 1,300 fewer tokens per turn across steps.
Figure 5 Prompt compression became safe only after a regression test exposed serialized agents and a one-sentence fix restored parallelism; the resulting token savings recur on every model turn.

Deliver results; don't make the agent fetch them twice

Agents often run independent work in the background — a long shell command beside a sub-agent investigation. Notifications let the agent proceed until work finishes. Previously, a completion notification did not include the result, forcing the agent to spend another turn retrieving output the harness already had. When several tasks finished close together, that detour repeated. Completion notifications are now batched, and results are delivered directly in the existing tool-result format. Explicit reads for work still running behave as before.

Before-and-after sequence diagram comparing orchestration behavior. Before: model waits on separate shell and sub-agent completions, causing retrieval detours and four LLM calls to process two results. After: a harness batches related completions and emits synthetic tool events so background work continues while waiting; both results are processed together in a single LLM call. The visual emphasizes reduced latency and fewer model round trips.
Figure 6 Before, each background completion could wake a retrieval-only model turn. After, the harness batches eligible completions and delivers completed results in the existing tool-result format.

Previously, each completed task required one model call to request its result and another to process it, meaning four model calls for a shell command plus sub-agent before work could continue. Now both completions arrive batched, so one model call processes everything. Removing those retrieval detours also avoids carrying full session context through unnecessary calls. Without compressing, summarizing, or withholding content, this reduced average token-related usage, measured in AI Credits, by about 2.3%.

What works in one workflow fails in another

A tighter set of file-tool instructions, inspired by positive results in Copilot code review, increased cost in a Copilot CLI online experiment and was not shipped. In contrast, removing line-number prefixes and selective output compression each reduced average prompt tokens per review by roughly 5% in independent evaluations across code review tasks using the production model, with no material change in tracked quality metrics. These results are separate from an earlier migration of code review to shared file tools which, combined with review-instruction tuning, reduced code review cost by about 20%. Changes must be measured where they run.

Rules for building efficient coding agents

  1. Optimize the completed task, not the tool call. Shorter output isn't cheaper if the agent spends extra turns recovering what was removed.
  2. Optimize orchestration, not just model output. Eliminate model turns for work the harness can finish deterministically.
  3. Compress by what the output represents. Preserve exact content, prefer lossless transformations, and track how often agents use the recovery path.
  4. Prompt rewrites can have unintended consequences. Validate that intended behaviors are preserved.
  5. Evidence is workload-specific. Re-evaluate changes in offline benchmarks, online experiments, and across every product surface.

None of these changes made the model smarter. They removed work the model never needed to do, including delivery of completed background work without retrieval detours and, across GitHub Copilot products using the same harness, selective compression, the unneeded line numbers, and the lighter prompt. The cumulative effect is fewer model turns, less redundant context, and more context budget devoted to actual engineering.