When an agent’s best tool is none of them
Building a reliable text-to-SQL agent is a classic exercise in over-engineering. Internal data teams know the pattern: specialized tools, strict prompt scaffolding, fine-tuned context windows, custom retrieval layers. Every edge case adds another patch. Every model update demands recalibration. Before long, maintaining the agent’s tooling becomes the real job.
The team behind Vercel’s internal data agent, d0, hit that wall. d0 was designed to translate natural-language Slack questions into SQL against the company’s Cube semantic layer, so anyone could query the warehouse without waiting for an analyst. The problem was getting out of its own way. The team reports that a radical simplification—replacing most of the agent’s machinery with a single bash tool—improved accuracy from ~80% to 100% and roughly tripled speed.
From walled garden to open shell
The original d0 architecture was built on a series of assumptions about model limitations: that it would get lost in a wide schema, hallucinate table names, or make bad joins. To counter that, the team added guardrails—multiple purpose-built tools for schema discovery and validation, hand-crafted prompts to bound reasoning, and careful pre-filtering of context. They also wrote retrieval logic to surface only the schema information they believed the model needed.
Looking back, the team realized they were doing the model’s thinking for it. Every patch added an edge case for the next patch to handle. The tools were trying to protect the agent from problems the model could already solve on its own.
The team’s solution was drastic reduction. The agent was stripped down to a single tool: execute arbitrary bash commands. This "file system agent" gets direct access to the Cube semantic layer files—YAML, Markdown, and JSON—and uses standard Unix utilities like cat, grep, find, and ls to navigate the data.
The architecture now looks like a minimal stack: Claude Opus 4.5 via the AI SDK for reasoning, a Vercel Sandbox for context exploration, Vercel Gateway for routing and observability, a Next.js API route with Vercel’s Slack Bolt for the chat interface, and Cube’s semantic layer served as plain files.
The argument is straightforward: the semantic layer files already contain complete definitions for dimensions, measures, joins, and relationships. The team spent months building tools to summarize information the model could simply grep for itself. By handing the agent raw files and Unix tools, they let an analyst’s most natural workflow—read, search, connect—become the agent’s workflow, too.
The numbers favor less tooling
Benchmarking the two architectures across five representative queries showed the file system agent winning every comparison. The old architecture’s worst case failed outright after 724 seconds, 100 steps, and 145,463 tokens. The new agent handled that same query in 141 seconds with 19 steps and 67,483 tokens—a 3.5x speedup, a ~37% token reduction, and a successful result where the old stack had failed.
Reliability also improved. The agent succeeded on every benchmark query, achieving a 100% success rate for the simple reason that fewer steps meant fewer failure points and less room for compounding error. The team also observed the model catching edge cases that weren’t anticipated by the engineers, and being able to trace the agent’s reasoning path from file reads to SQL more clearly than before.
Architecture is a bet on the model
The team draws several lessons from the rebuild. First: don’t fight the abstractions you already have. The filesystem is the product of 50 years of refinement. grep still does the job; custom search tools often just recreate it with more maintenance burden removed.
Second: question tooling assumptions against current models. The choice to constrain the agent’s workflow reflected a lack of confidence in the earlier model. With Claude Opus 4.5’s improved reasoning, those constraints became liabilities. They now believe every tool is a decision the engineer is placing in front of the model. It’s often better to let the model decide how to carry out the task. The tradeoff only works, though, if the model’s context is well-organized and well-documented in the first place. Raw file access to unstructured, poorly named files would yield faster bad queries, not better analytics.
Third, they suggest: start minimal. The impulse among agent builders is to plan for every edge case, but that’s often speculative. Begin with a model, a filesystem, and a goal. Add complexity only after the minimalist version has demonstrated the need. Models are improving far faster than tooling can keep up; agent architectures should position themselves for the model, not lock in a particular inferior one’s needs.
