The harness matters as much as the model

In agentic coding, the model supplies the raw capability, but the harness determines how effectively that capability is applied. GitHub Copilot’s agentic harness is a shared component of the GitHub Copilot SDK that powers the Copilot CLI, GitHub Copilot app, Copilot code review, and a range of other experiences across GitHub and Microsoft. Because the harness is common infrastructure, improvements benefit every surface that relies on it.

The harness orchestrates the tools, context, and workflow around the model. For developers, that means it needs to be fast, predictable, and economical with tokens. The benchmark results below show how the GitHub Copilot agentic harness measures up on efficiency and performance across a range of agentic software engineering tasks.

Benchmarking approach

Evaluation of the Copilot agentic harness combines public industry-standard benchmarks with internal benchmarks derived from large codebases at GitHub and Microsoft, supplemented by real-world metrics and online experiments. To isolate harness performance, the evaluation controls for model, benchmark task, context window, reasoning effort, tool selection, and MCP servers.

Results are reported for four leading models: Claude Sonnet 4.6, Claude Opus 4.7, GPT-5.4, and GPT-5.5. Throughout the evaluation, GitHub Copilot CLI is compared against the model-vendor harnesses that ship those models natively: Claude Code for the Claude models and Codex CLI for the GPT models.

Benchmark Domain Purpose 
SWE-bench Verified 500 human-validated bug-fix tasks from open-source Python repositories Established industry-standard benchmark for coding agents 
SWE-bench Pro More difficult, multi-step engineering tasks requiring deeper reasoning and broader code changes Better reflects complex, real-world software engineering work 
SkillsBench How effectively an agent uses skills to solve tasks Evaluates extensibility and skill use and triggering capabilities 
TerminalBench Agent performance on terminal-based tasks Measures effectiveness in command-line workflows used by developers 
Win-Hill Internal benchmark for tasks running inside Windows containers Validates that performance generalizes across operating systems and environments 

Token efficiency

Holding the model and the task fixed across multiple benchmarks, the GitHub Copilot harness achieves task completion rates on par with model-vendor harnesses while consuming fewer tokens in most configurations.

Chart showing Copilot CLI versus model-vendor harnesses using SWE-bench Verified, SWE-bench Pro, SkillsBench, Win-Hill, and TerminalBench2 tests. For Sonnet 4.6 and Opus 4.7, Copilot CLI performed better in all cases, using fewer tokens. For GPT 5.4 and GPT 5.5, CLI performed better in all cases except SWE-bench Verified, where it did 7% and 4% worse, respectively.
Token efficiency: GitHub Copilot CLI vs. other model-vendor harnesses

Task resolution

Token efficiency only matters if the work actually gets done. Resolution rates for the GitHub Copilot agentic harness are on par with model-vendor harnesses when evaluated with a fixed model and benchmark task. The effective parity indicates that the underlying model’s full potential is accessible through the Copilot harness, with differences either direction within the variance of stochastic model runs.

Task resolution benchmarking test results for Copilot CLI versus model-vendor harnesses. For SWE-bench Verified tests, Copilot CLI performed better with Sonnet 4.6 and Opus 4.7, but worse with GPT 5.4 and GPT 5.5. For SWE-bench Pro, Copilot CLI only performed slightly worse with Sonnet 4.6, and performed better for other models. For SkillsBench, Copilot CLI performed worse for Sonnet 4.6 and Opus 4.7, but better for GPT models. For Win-Hill, Copilot CLI performed equal or better for all models. For TerminalBench 2, Copilot CLI performed better for Sonnet 4.6 and Opus 4.7, equal for GPT 5.5, and worse for GPT 5.4.
Task resolution: GitHub Copilot CLI vs. the model-vendor harnesses

Variance analysis on TerminalBench 2.0

Beyond aggregate results, run-to-run variance is worth examining closely. A variance analysis on TerminalBench 2.0 illustrates task completion, cost per task, and the reproducibility of each agent-model configuration.

A diagram showing mean cost per task compared to the resolution rate. Copilot CLI performs equal to or better than model-vendor harnesses.
Resolution rate vs. cost per task. Up and to the left is better: solve more, spend less. 

Each marker represents one agent-and-model configuration on TerminalBench 2.0. The vertical axis shows resolution rate and the horizontal axis shows dollar cost per task. The shaded ellipse around each point marks the ±1σ spread across runs, showing how much each configuration varies between runs.

Three observations stand out:

  1. Copilot is on par or ahead on both axes. The purple Copilot markers sit within overlapping ellipses with their same-model competitors for nearly every model. Copilot is never below a competitor on task completion, nor to the right on cost.
  2. Run-to-run variability exists across all configurations. Each agent-model combination was run at least five times. Tighter ellipses indicate more reproducible results; wider ellipses show results that swing further from run to run on cost and completion.
  3. Model choice presents a trade-off. The GPT models (left side) deliver the best value with strong resolution at the lowest cost. Claude Opus (upper right) reaches the highest resolution at a premium. Copilot supports both options, allowing users to select for efficiency or peak quality per task.

A multi-model harness

The GitHub Copilot agentic harness supports 20+ frontier models across GPT, Claude, Gemini, and MAI families, plus bring-your-own-key for open-source and local models. Users can select a model per task based on capability and cost, or rely on Auto model selection to balance task intent and model health for token optimization.

A multi-model architecture also enables harness-level capabilities that a vendor-specific harness cannot provide. Rubber Duck uses cross-model-family critique, where one model reviews another model’s work to improve outcomes beyond what a single model achieves on its own.

Methodology

All agents were run with equivalent settings across models, tasks, and environments to keep comparisons controlled and reproducible. Each run had a two-hour timeout, with agents running non-interactively in single-turn mode, web tools disabled, and all other tools allowed.

TerminalBench 2.0 analysis: Default settings were used for agents with reasoning effort set to medium, including tool search enabled for Claude Code and the github-mcp-server for Copilot CLI. Codex and Claude Code used direct Anthropic and OpenAI endpoints. All 89 TerminalBench 2.0 tasks produced results, with infrastructure failures re-run to completion; model-generated errors were retained in the analysis. Each model was evaluated across five independent runs, and Copilot was tested in two separate batches to enable direct comparison with Claude Code and Codex.

All other benchmarks: Agent-model pairs were normalized to the same context window size, prompt token limits, and reasoning effort (medium), with no tool search and no MCP servers. Harness default built-in tools were kept. Infrastructure-related anomalies and network-access effects were excluded uniformly across all agents for each benchmark. For smaller benchmarks (fewer than 100 instances), five independent runs were conducted and the best-scoring run was reported. All metrics are pass@1. These normalizations differ from public benchmark submissions, which typically use higher reasoning effort and other tuned settings.