A Field Report on Agent-First Development
Software engineers have a long tradition of building systems to eliminate their own toil, only to end up maintaining those systems for everyone else. Recently, an AI researcher on the Copilot Applied Science team took this a step further, automating away not just repetitive tasks but the intellectual work of analyzing coding agent performance. The result was eval-agents, a project built with GitHub Copilot as the primary contributor, and a set of lessons about what it takes to make agent-driven development work at scale.
Origins: The Trajectory Analysis Problem
The project began with a concrete bottleneck: evaluating coding agents against standardized benchmarks like TerminalBench2 or SWEBench-Pro. Each task in an evaluation dataset produces a trajectory—a .json file containing the agent's thought process and actions. With dozens of tasks per dataset, and many benchmark runs to analyze daily, the volume reaches hundreds of thousands of lines of code.
The initial workflow relied on GitHub Copilot to surface patterns in these trajectories, reducing the number of lines needing human review from hundreds of thousands to a few hundred. But the process was still repetitive. The solution was to build agents to automate the analysis itself.
Design Goals for Collaborative Agents
The project pursued three goals from the outset:
- Make the agents easy to share and use.
- Make it easy to author new agents.
- Make coding agents the primary vehicle for contributions.
The first two goals align with standard open-source practices. The third proved the most influential, shaping the entire development approach. The practical setup used Copilot CLI as the coding agent, Claude Opus 4.6 as the model, and VSCode as the IDE, with the Copilot SDK providing access to existing tools, MCP servers, and the ability to register new tools and skills.
Strategy 1: Prompting Like a Collaborator
Coding agents excel at well-scoped problems but require guidance on complex ones. The effective pattern is treating the agent like a senior engineer: provide extended context, explain assumptions, and use planning modes before agent modes.
A concrete example involved writing a prompt to add robust regression tests. The prompt included unstructured thinking about the problem, which led to a back-and-forth with Copilot that eventually produced guardrails akin to contract testing—protected so that only humans can update them. The conversational, verbose approach consistently outperformed terse problem statements.
Strategy 2: Architecture as an Agent Enabler
The traditional trade-off between refactoring and shipping new features disappears in agent-first development. Clean architecture, thorough documentation, and comprehensive tests become the highest-value work because they directly enable the coding agent to navigate the codebase effectively.
Much of the development time on eval-agents went into refactoring file structures, documenting patterns, and adding tests for edge cases discovered along the way—including cleaning up dead code left by the agents themselves. This maintenance work pays off because it allows Copilot to operate with the same understanding a human engineer would have. The luxury of asking "Knowing what I know now, how would I design this differently?" and acting on the answer becomes feasible when the agent can handle the rearchitecture.
Strategy 3: Blameless Process for Agents
The industry's shift from "trust but verify" to "blame process, not people" applies directly to agent-driven development. Instead of being skeptical of every agent action, the focus is on building guardrails that prevent mistakes—and, when mistakes happen, adding new guardrails so they cannot recur.
This translates into concrete engineering practices: strict typing forces the agent to conform to interfaces; robust linters impose implementation rules; and contract, integration, and end-to-end tests—now much cheaper to build with agent assistance—verify that new changes do not break existing features. When Copilot has these tools in its development loop, it can check its own work continuously.
A Working Development Loop
The resulting workflow is structured around four stages:
- Plan: Start with Copilot using
/plan, iterating on the plan until testing and documentation updates are included. Docs should be written before code, serving as additional guidelines. - Implement: Let Copilot execute the feature on
/autopilot. - Automated review: Prompt Copilot to request a review from the Copilot Code Review agent, address relevant comments, and re-request until no issues remain.
- Human review: The final check, enforcing architectural and quality patterns.
Beyond the feature loop, regular maintenance prompts keep the codebase agent-ready:
/plan Review the code for any missing tests, any tests that may be broken, and dead code/plan Review the code for any duplication or opportunities for abstraction/plan Review the documentation and code to identify any documentation gaps. Be sure to update the copilot-instructions.md to reflect any relevant changes
These can run automatically on a weekly basis, or on demand as features land.
The Numbers and the Takeaway
The results of this approach are striking. Five first-time contributors joined the project and, in under three days, created 11 new agents, four new skills, and introduced the concept of eval-agent workflows—spanning a change of +28,858/-2,884 lines of code across 345 files.
What began as a frustration with repetitive manual analysis ends with a fundamental shift in software development practice. The agent-first mindset forces prioritization of the engineering fundamentals: clean architecture, thorough documentation, meaningful tests, and thoughtful design. Onboard an agent well, provide clear context, and build guardrails so its mistakes do not become disasters—then trust it to expand its capabilities. The technology is new, but the principles are not.
