Coding Agents Are a Different Category of Tool
Anyone who has tried Claude Code, Gemini Code, or Simon Willison's LLM CLI has encountered something distinct from a chatbot or autocomplete engine. These agents read your codebase, run test suites, search documentation, and modify files asynchronously. The best way to understand what makes them work is to build one. We assembled our own CLI coding agent using the Pydantic-AI framework and the Model Context Protocol (MCP). While our implementation ran on AWS Bedrock, Pydantic-AI also supports other mainstream providers and fully local LLMs.
Commercial tools are built for general use cases. Our agent was customized to our internal development standards around testing, documentation, code reasoning, and file system operations. It captured the eccentricities of our specific project context—and building it provided direct insight into how these systems function and where our own GenAI tooling could improve.
High-Level Agent Architecture
Our coding assistant is composed of several core components:
- Core AI model: Claude from Anthropic, accessed via AWS Bedrock
- Pydantic-AI framework: The agent framework and base utilities
- MCP servers: Independent processes providing the agent with specialized tools via a standardized interface
- CLI interface: The user-facing interaction layer
The Model Context Protocol (MCP) lets the model use tools from multiple servers through one standard.
Adding a new capability simply means implementing another MCP server.
The Foundation
We started by creating a basic project structure and installing dependencies:
uv init uv add pydantic_ai uv add boto3
The primary dependencies were pydantic-ai for building the agent and boto3 for AWS API interactions. Claude Sonnet 4 was selected for its code understanding and generation capabilities. The configuration in main.py:
import boto3 from pydantic_ai import Agent from pydantic_ai.mcp import MCPServerStdio from pydantic_ai.models.bedrock import BedrockConverseModel from pydantic_ai.providers.bedrock import BedrockProvider
bedrock_config = BotocoreConfig(
read_timeout=300,
connect_timeout=60,
retries={"max_attempts": 3},
)
bedrock_client = boto3.client(
"bedrock-runtime", region_name="eu-central-1", config=bedrock_config
)
model = BedrockConverseModel(
"eu.anthropic.claude-sonnet-4-20250514-v1:0",
provider=BedrockProvider(bedrock_client=bedrock_client),
)
agent = Agent(
model=model,
)
if __name__ == "__main__": agent.to_cli_sync()
This yields a fully working CLI with chat functionality. Useful, but limited—so we began adding capabilities.
First Capability: Running the Tests
Instead of running tests manually after each iteration, we gave the agent the same pytest command used in the terminal:
import subprocess
@agent.tool_plain()
def run_unit_tests() -> str:
"""Run unit tests using uv."""
result = subprocess.run(
["uv", "run", "pytest", "-xvs", "tests/"], capture_output=True, text=True
)
return result.stdout
This changed the workflow significantly. Saying "X isn't working" would prompt the agent to:
- Run the test suite
- Identify the specific failing tests
- Analyze the error messages
- Suggest targeted fixes
We avoided copying and pasting terminal output into a separate chat. The agent gained relevant context about the current state of the codebase automatically.
Steering with Instructions
We observed something problematic: the agent sometimes "fixed" failing tests by modifying the tests themselves rather than the implementation. Setting clearer guidance was necessary:
instructions = """ You are a specialised agent for maintaining and developing the XXXXXX codebase. ## Development Guidelines: 1. **Test Failures:** - When tests fail, fix the implementation first, not the tests - Tests represent expected behavior; implementation should conform to tests - Only modify tests if they clearly don't match specifications 2. **Code Changes:** - Make the smallest possible changes to fix issues - Focus on fixing the specific problem rather than rewriting large portions - Add unit tests for all new functionality before implementing it 3. **Best Practices:** - Keep functions small with a single responsibility - Implement proper error handling with appropriate exceptions - Be mindful of configuration dependencies in tests Remember to examine test failure messages carefully to understand the root cause before making any changes. """
agent = Agent( instructions=instructions, model=model, )
With explicit instructions about Test Driven Development and minimal changes, the agent stopped suggesting large refactors where a small fix was appropriate. Building everything from scratch and continuously tuning prompts, however, is not an efficient path. It was time to use protocols other people had already built.
Pluggable Capabilities via MCP
MCP is an open protocol that standardizes how applications provide context to LLMs. Think of MCP like a USB-C port for AI applications. Just as USB-C provides a standardized way to connect your devices to various peripherals and accessories, MCP provides a standardized way to connect AI models to different data sources and tools.
Servers run as a local process interacting over STDIN/STDOUT, with no data sharing. Each added server gave the agent capabilities that moved it closer to commercial CLI agents.
Sandboxed Python Execution
Large language models doing arithmetic by token prediction is unreliable, and executing arbitrary model-generated code in the main environment is potentially dangerous. The first MCP addition was Pydantic AI's default server for sandboxed Python execution:
run_python = MCPServerStdio(
"deno",
args=[
"run",
"-N",
"-R=node_modules",
"-W=node_modules",
"--node-modules-dir=auto",
"jsr:@pydantic/mcp-run-python",
"stdio",
],
)
agent = Agent(
...
mcp_servers=[
run_python
],
)
In this isolated environment, the agent could test ideas, prototype solutions, and verify its own suggestions. This is notably different from running tests locally—that requires the application environment. The sandbox makes calculations robust, scalable, and repeatable; frontier labs themselves use this approach over next-token generation for numerical work.
Mathematical operations, date calculations, and counts become significantly more reliable, and the agent can run a rapid iterate-verify loop on small Python snippets.
Current Library Documentation
Foundation models are trained on historical data, so their knowledge has a fixed cutoff that libraries and languages quickly surpass. Adding Context7 gave the agent access to up-to-date Python library documentation in LLM-consumable format:
context7 = MCPServerStdio(
command="npx", args=["-y", "@upstash/context7-mcp"], tool_prefix="context"
)
Rather than relying on stale training data for newer or advanced features, the agent could consult current documentation. This restored reliability in real development workflows.
AWS-Specific Context
Because the agent was built for an AWS platform, the AWS Labs MCP servers added comprehensive documentation access:
awslabs = MCPServerStdio(
command="uvx",
args=["awslabs.core-mcp-server@latest"],
env={"FASTMCP_LOG_LEVEL": "ERROR"},
tool_prefix="awslabs",
)
aws_docs = MCPServerStdio(
command="uvx",
args=["awslabs.aws-documentation-mcp-server@latest"],
env={"FASTMCP_LOG_LEVEL": "ERROR", "AWS_DOCUMENTATION_PARTITION": "aws"},
tool_prefix="aws_docs",
)
Mentioning that "Bedrock is timing out" or "model responses are truncated" now led the agent to AWS documentation troubleshooting guides. the AWS Labs MCP collection—covering CloudWatch metrics, Lambda debugging, IAM policy analysis, and more—offers far beyond what we've explored, but documentation access alone made cloud debugging extremely conversational.
General Search and Structured Reasoning
Issues often attract Stack Overflow threads and GitHub discussions before any official docs. A general internet search tool was added:
internet_search = MCPServerStdio(command="uvx", args=["duckduckgo-mcp-server"])
For obscure errors or breaking ecosystem changes, the agent searched current discussions and solutions—finding recent deployment concerns and dependency migration topics rapidly.
One of the most impactful additions was a code reasoning MCP. Complex problems are processed systematically instead of acting on the first apparent approach:
code_reasoning = MCPServerStdio(
command="npx",
args=["-y", "@mettamatt/code-reasoning"],
tool_prefix="code_reasoning",
)
Only a shallow explanation of its analysis where potential. Facing an intermittently failing API call, an agent would return a structured taxonomy of probable causes instead of scattered guesses.
Patience for Analyzing, Not Generating
As capabilities grew, reasoning tasks slowed noticeably—especially when responses were not correctly formatted on the first attempt. To accommodate these deeper analytical operations, the Bedrock configuration was made more tolerant:
bedrock_config = BotocoreConfig(
read_timeout=300,
connect_timeout=60,
retries={"max_attempts": 3},
)
bedrock_client = boto3.client(
"bedrock-runtime", region_name="eu-central-1", config=bedrock_config
)
Lengthier timeouts allowed the agent to reason through complex tasks without cutting off. Analyzing larger codebases or architectural trade-offs took as long as necessary to produce a well-reasoned answer rather than a premature one.
From Assistant to Operator
With the core agent in place, adding a Desktop Commander MCP server changes the nature of the tool. The agent is no longer limited to reasoning and code execution—it can now act directly on the development environment.
desktop_commander = MCPServerStdio(
command="npx",
args=["-y", "@wonderwhy-er/desktop-commander"],
tool_prefix="desktop_commander",
)
Desktop Commander bundles a wide range of capabilities: file operations, terminal commands with process management, targeted code edits via edit_block, and interactive REPL sessions. It builds on the MCP Filesystem Server but adds practical extras like search-and-replace editing and process control.
The practical effect is significant. A request like "the authentication tests are failing, please fix the issue" triggers a full workflow:
- Run the test suite to see the specific failures
- Read the failing test files to understand what was expected
- Examine the authentication module code
- Search the codebase for related patterns
- Look up the documentation for the relevant library
- Make edits to fix the implementation
- Re-run the tests to verify the fix
- Search for similar patterns elsewhere that might need updating
All of this happens in a single conversation thread with context retained throughout. The agent is not generating isolated suggestions; it is debugging, editing, and verifying fixes as a collaborative partner. The security model is configurable with allowed directories, blocked commands, and permission boundaries. Full details are available in the Desktop Commander documentation.
The Assembled System
The final agent configuration brings together all these tools:
import asyncio
import subprocess
import boto3
from pydantic_ai import Agent
from pydantic_ai.mcp import MCPServerStdio
from pydantic_ai.models.bedrock import BedrockConverseModel
from pydantic_ai.providers.bedrock import BedrockProvider
from botocore.config import Config as BotocoreConfig
bedrock_config = BotocoreConfig(
read_timeout=300,
connect_timeout=60,
retries={"max_attempts": 3},
)
bedrock_client = boto3.client(
"bedrock-runtime", region_name="eu-central-1", config=bedrock_config
)
model = BedrockConverseModel(
"eu.anthropic.claude-sonnet-4-20250514-v1:0",
provider=BedrockProvider(bedrock_client=bedrock_client),
)
agent = Agent(
model=model,
)
instructions = """
You are a specialised agent for maintaining and developing the XXXXXX codebase.
## Development Guidelines:
1. **Test Failures:**
- When tests fail, fix the implementation first, not the tests
- Tests represent expected behavior; implementation should conform to tests
- Only modify tests if they clearly don't match specifications
2. **Code Changes:**
- Make the smallest possible changes to fix issues
- Focus on fixing the specific problem rather than rewriting large portions
- Add unit tests for all new functionality before implementing it
3. **Best Practices:**
- Keep functions small with a single responsibility
- Implement proper error handling with appropriate exceptions
- Be mindful of configuration dependencies in tests
Remember to examine test failure messages carefully to understand the root cause before making any changes.
"""
run_python = MCPServerStdio(
"deno",
args=[
"run",
"-N",
"-R=node_modules",
"-W=node_modules",
"--node-modules-dir=auto",
"jsr:@pydantic/mcp-run-python",
"stdio",
],
)
internet_search = MCPServerStdio(command="uvx", args=["duckduckgo-mcp-server"])
code_reasoning = MCPServerStdio(
command="npx",
args=["-y", "@mettamatt/code-reasoning"],
tool_prefix="code_reasoning",
)
desktop_commander = MCPServerStdio(
command="npx",
args=["-y", "@wonderwhy-er/desktop-commander"],
tool_prefix="desktop_commander",
)
awslabs = MCPServerStdio(
command="uvx",
args=["awslabs.core-mcp-server@latest"],
env={"FASTMCP_LOG_LEVEL": "ERROR"},
tool_prefix="awslabs",
)
aws_docs = MCPServerStdio(
command="uvx",
args=["awslabs.aws-documentation-mcp-server@latest"],
env={"FASTMCP_LOG_LEVEL": "ERROR", "AWS_DOCUMENTATION_PARTITION": "aws"},
tool_prefix="aws_docs",
)
context7 = MCPServerStdio(
command="npx", args=["-y", "@upstash/context7-mcp"], tool_prefix="context"
)
agent = Agent(
instructions=instructions,
model=model,
mcp_servers=[
run_python,
internet_search,
code_reasoning,
context7,
awslabs,
aws_docs,
desktop_commander,
],
)
@agent.tool_plain()
def run_unit_tests() -> str:
"""Run unit tests using uv."""
result = subprocess.run(
["uv", "run", "pytest", "-xvs", "tests/"], capture_output=True, text=True
)
return result.stdout
async def main():
async with agent.run_mcp_servers():
await agent.to_cli()
if __name__ == "__main__":
asyncio.run(main())
This combination reshapes the daily workflow in several distinct ways:
- Debugging is collaborative—the agent analyzes errors, proposes hypotheses, and helps verify solutions.
- Learning accelerates with unfamiliar libraries, as the agent explains existing code and the reasoning behind certain approaches.
- Context switching drops, since documentation, Stack Overflow, the AWS Console, and the IDE are all reachable from one interface that retains the problem's context.
- Problem-solving becomes structured, with complex issues broken into logical steps and alternatives explored explicitly.
- Code review happens pre-commit, with the agent flagging potential issues and suggesting improvements.
Lessons From Building a CLI Agent
Constructing this agent highlighted several principles about the current state of the paradigm:
- MCP integration is the core enabler. No single capability is transformative, but the combination—running tests, reading files, searching documentation, executing code, accessing AWS services, and reasoning—creates a qualitatively different tool.
- Current information matters. Real-time search and up-to-date documentation keep the agent reliable for real-world work where training data goes stale.
- Structured reasoning elevates the agent beyond autocomplete into a partner that can decompose problems and weigh alternatives.
- Context is the differentiator. Commercial agents are effective largely because they maintain state across tools; a useful agent must remember what it learned from a test run when it later edits files.
- Specialization has value. An agent tailored to a specific codebase outperforms general tools because it understands the project's patterns and conventions, and can be modified when it falls short.
Where This Is Heading
Development in this space is moving quickly. Areas worth watching include:
- AWS-specific tooling: the AWS Labs MCP servers offer deep integration for cloud-native work, from CloudWatch metrics to Lambda debugging and IAM policy analysis.
- Workflow automation: training agents on routine development processes for end-to-end handling, and connecting them to project management tools for context on priorities.
- Benchmarking: Terminal Bench provides a dataset and leaderboard for measuring a custom agent against commercial offerings.
CLI coding agents represent a shift from AI as a writing aid to AI as a development partner. Unlike autocomplete or Q&A systems, these agents understand the full project context, execute tasks across multiple tools, maintain state through complex workflows, and learn from a specific codebase.
Building one yourself, even a simple version, is the clearest way to understand the trajectory of the technology and to make informed use of commercial tools as they mature. The future of development is less about writing code faster and more about having an intelligent collaborator that understands goals, constraints, and code well enough to help think through problems and implement solutions together. The most direct path to that understanding is to build it.
