Why GitHub MCP Server needs offline evaluation

MCP (Model Context Protocol) standardizes how LLMs connect to APIs and data. An MCP server publishes a list of available tools, their descriptions, and required parameters, letting a model invoke the right function for a user request. The GitHub MCP Server powers many GitHub Copilot workflows, both in and outside of GitHub.

For the engineering team maintaining it, small changes can have outsized effects. Tightening a tool description, merging similar tools, or dropping one entirely can shift a model's behavior. When descriptions are ambiguous, agents pick the wrong tool, skip steps, or pass malformed arguments. The team needed a way to iterate safely—knowing whether a change improved performance before users ever see it.

Offline evaluation provides that safety net. It catches regressions early, shortens the feedback loop, and lets the team ship changes that genuinely improve results.

The evaluation pipeline in three stages

The GitHub MCP team maintains curated benchmark datasets to test how well tool prompts work across different models. Each benchmark contains three parts:

  1. Input: a user request in natural language
  2. Expected tools: the tools the model should call
  3. Expected arguments: the arguments each tool should receive

A few examples from the dataset:

Input: How many issues were created in the github/github-mcp-server repository during April 2025?
Expected tools: list_issues with arguments:

owner: github 
repo: github-mcp-server 
since: 2025-04-01T00:00:00Z

Input: Merge PR 123 in github/docs using squash merge with title "Update installation guide"
Expected tools: merge_pull_request with arguments:

owner: github
repo: docs 
pullNumber: 123 
merge_method: squash 
commit_title: Update installation guide

Input: Request reviews from alice456 and bob123 for PR 67 in team/project-alpha
Expected tools: update_pull_request with arguments:

owner: team 
repo: project-alpha 
pullNumber: 67
reviewers: ["alice456", "bob123"]

Input: Summarize the comments in discussion 33801, in the facebook/react repository
Expected tools: get_discussion_comments with arguments:

owner: facebook
repo: react
discussionNumber: 33801

The pipeline runs in three phases:

  • Fulfillment: each benchmark runs across multiple models, with the full list of available MCP tools provided alongside every request. The run records which tools the model invoked and the arguments it supplied.
  • Evaluation: raw outputs are processed into metrics and scores.
  • Summarization: dataset-level statistics are aggregated into the final report.

Measuring tool selection

When a benchmark expects a single tool call, tool selection becomes a multi-class classification problem. Each tool is a class; each benchmark is labeled with the expected tool. The team evaluates classifications with accuracy, precision, recall, and F1-score.

  • Accuracy is the share of inputs that produced the expected tool call, computed across the whole dataset.
  • Precision measures how often a called tool was the correct one. Low precision means the model calls the tool even when another tool is expected.
  • Recall measures how often an expected tool was actually called. Low recall suggests the model misses the tool entirely or substitutes another one.
  • F1-score is the harmonic mean of precision and recall.

Confusion between similar tools drives these metrics down. A known example is list_issues and search_issues. Say the dataset has 10 benchmarks for each, and list_issues is correctly called in all 10 of its cases, but also in 3 of the search_issues cases:

Precision (list_issues) = 10 / (10 + 3) = 0.77

Recall (search_issues) = 7 / 10 = 0.7

The confusion matrix below visualizes exactly where these errors happen:

Expected tool / Called toolsearch_issueslist_issues
search_issues73
list_issues010

That visibility lets the team pinpoint which tool descriptions need refinement to minimize mix-ups.

Argument correctness metrics

Calling the right tool is only half the job; the model also needs to pass correct arguments. The team tracks four argument-quality metrics, computed only for correctly selected tools:

  • Argument hallucination: how often the model supplies an argument name not defined for the tool
  • All expected arguments provided: whether every expected argument is present
  • All required arguments provided: whether all required arguments are included
  • Exact value match: whether supplied values match expected values exactly

The final report summarizes each tool's performance across all four metrics.

Known gaps and planned improvements

The current framework has clear limitations. Benchmark volume is the biggest one: with so many tools, per-tool coverage needs expansion. Judgments based on a handful of examples aren't dependable on their own, so the team is adding more benchmarks to strengthen classification evaluations.

The pipeline also only handles single tool calls. Real users often trigger sequences, where later tool calls depend on the output of earlier ones. Evaluating those flows requires executing tools or mocking responses during evaluation, rather than just fetching the MCP tool list.

Summarization needs an update as well. Treating tool selection as multi-class classification assumes one tool per input. For flows where one input triggers several tools, multi-label classification is the better model.

The payoff of this work is steadier: fewer regressions, clearer diagnosis of failures, and more reliable agents built on GitHub MCP Server.