Why Traditional Tests Fall Short for AI

Software built on large language models behaves differently from conventional code. A deterministic function returns the same result for the same input every time; an LLM is probabilistic, producing new variations with each call. That unpredictability makes traditional unit and integration tests insufficient for judging whether an AI system is actually good at its job. It is less like verifying that two plus two equals four and more like forecasting tomorrow's weather.

At Vercel, building AI products such as v0 has required a different quality framework. The approach the team has settled on is eval-driven development. Evals function as end-to-end tests for probabilistic systems, measuring output quality against defined criteria through automated checks, human judgment, and AI-assisted grading. They don't test individual code paths; they gauge overall performance.

Three primary types of evals exist:

  • Code-based grading: Automated checks work well for objective criteria and fast feedback, such as verifying that output contains a keyword or matches a regular expression. Not every judgment can be coded, however.
  • Human grading: Subjective qualities like clarity, coherence, and creativity need human judgment. This is the most reliable route for nuanced assessments.
  • LLM-based grading: Using another model to rank output scales well for complex judgments and costs significantly less than human review, though it is less trustworthy. Note that LLM evals still run 1.5x to 2x more expensive than code-based grading.

Search engineers faced a similar shift two decades ago. The web's scale and the unpredictability of user queries made exhaustive hard-coded testing pointless. Teams that owned search quality moved to eval-centric processes, accepting that any change brought both improvements and regressions. Slower and more subjective than unit tests, that approach proved most effective in a complex, non-deterministic environment.

A Concrete Example: React Component Generation

Consider an assistant that generates React components from text descriptions. Say the system prompt asks for a functional arrow component accepting an array of strings, rendering each in a list, styled with Tailwind CSS. The expected output and an actual output may differ in ways that matter:

  • Expected output uses Tailwind classes (list-none p-0, font-bold); actual uses inline styles.
  • Expected output includes a key prop on list items; actual output omits it.
  • Expected output is an arrow function; actual is a traditional function declaration.

The AI-generated code is functional, but quality and adherence to conventions are off. Evals can surface those discrepancies on multiple axes. A code-based check might regex for inline styles or verify key props exist; human grading can inspect the output and judge it with a simple thumb up or down; an LLM evaluator can compare the generated code against the expected output. Whatever method is used, it must yield a clear score per criterion so the team can tell whether the system is improving or regressing. Those scores then guide fine-tuning and prompt refinement, closing a loop that makes evals far more potent than a simple test replacement.

The AI-Native Flywheel

Evaluations become the engine of a continuous cycle when paired with high-quality data, durable strategies, and real user feedback. Vercel calls this loop the AI-native flywheel. A robust feedback cycle lets a team rely on data rather than intuition, pushing an AI system toward consistent improvement.

Evals as the Foundation

Evals expose strengths and weaknesses, steering development. The difficulty is running them at scale. Ad-hoc review or generic benchmarks lack the specificity needed for targeted fixes. Tooling like Braintrust, which automates evaluations with LLM-based graders and heuristics, can fold production logs and real user interactions into the evaluation data, closing the gap between lab and live use.

Data Gaps and Golden Data

Sound evals also reveal where data is thin. High-quality data fuels effective models — garbage in, garbage out still applies. Golden data gives leverage for fine-tuning, and every new data source needs its own evals to confirm the AI uses it correctly without drifting into hallucination.

Model and Strategy Selection

Model offerings change constantly, and evals make it possible to test new options against the same criteria. That enables quick identification of the best model-plus-data-plus-prompt combination for accuracy, latency, and cost. The AI SDK (npm i ai) simplifies the switching itself, offering a type-safe abstraction over multiple providers.

Feedback in All Its Forms

Continuous optimization depends on diverse signals from real users:

  • Explicit feedback: Let users rate output with thumbs up/down, stars, or a form.
  • Implicit feedback: Watch whether users rephrase or abandon interactions, a possible sign of misunderstanding.
  • Error reporting: Capture unexpected behavior to identify bugs and limitations.

Feeding these signals back into the eval suite turns user behavior into a driver of improvement. The product keeps adjusting as circumstances change.

How v0 Practices Eval-Driven Development

Vercel's own AI flagship, v0, was built on this philosophy. Its multi-layered suite mixes fast code checks, human grading by both end users and the team, and LLM-based grading for volume. All three flow back into the eval set, catching errors early and speeding up iterations. The team updates prompts nearly every day, leaning on evals to catch regressions and to verify accuracy when RAG content is refreshed.

Refusal and safety evals have a 100% pass rate, a priority for the product. Failing prompts are added to the suite so the system has to beat them to improve, a tactic that also applies to new capabilities like Vue or Python generation. Internal use of the product yields further feedback that the team can chase. Code-side graders verify formatting, imports, multi-file correctness, and a sensible balance between comments and actual code to correct "lazy" LLM output. An automated script reports pass/fail rates and trends, Braintrust stores details for manual review, and every pull request that could alter output generation lands with eval results attached.

Keeping evals current is a known challenge, especially since the system evolution obliges the suite to track it. The aim is to make the suite more sustainable and less hand-maintained without replacing expert review entirely.

Evaluation as a Development Method

Eval-driven development changes what quality control looks like for software with a probabilistic core. A predictable future for AI-native development is one where rigorous feedback loops — automated, human, and model-assisted — keep systems around a moving target: beneficial behavior measured against the things users actually care about.