Benchmarking agents on real Stripe work

LLMs have gotten good at solving scoped coding problems—function implementations, file-level refactors. But that skill doesn't necessarily translate to long-horizon engineering projects, which demand planning, state management, and recovery from failure. Even for an API designed for ease of use like Stripe's, shipping an end-to-end integration involves glue work across backend APIs, frontend code, and databases.

Stripe set out to measure whether AI agents can autonomously build complete integrations. The bar is unforgiving: for payments, a mostly correct integration is a failure. The real question isn't just whether an agent can generate code, but whether it can verify and test that code with the rigor of a human engineer.

The resulting Stripe integration benchmark is one of the first agentic development benchmarks for APIs in a production-realistic environment. It's designed to answer how well models understand the Stripe API, whether they can author correct backend and frontend code, and whether they can handle end-to-end tasks like updating packages, managing database state, and verifying integrations in a browser.

Building realistic integration environments

Working with product and technical support engineers, the team identified real-world integration scenarios—migrating Checkout flows, modeling business logic with Billing APIs—and built 11 diverse environments around them. Each evaluation has three components:

  • Environment: a full codebase with databases and scripts representing a realistic starting repository, plus test Stripe API keys for the agent to use during testing and submission validation.
  • Graders: automated components that score submissions. Most use deterministic tests exercising the finished software via API calls or automated UI tests. Some graders also inspect created Stripe API objects to validate artifacts—for instance, after a test payment, verifying a corresponding test-mode Checkout Session object exists.
  • Agent harness: a consistent runtime for all model evaluations, built on goose with a Model Context Protocol (MCP) server providing terminal, browser, and Stripe-specific search tools.

Eval schematic and representative snippet from an agent execution transcript

Building these environments required effort comparable to shipping real software. They needed realistic complexity—multiple files, scripts, databases—while remaining structured enough for clean, unambiguous, replicable grading. The tasks fall into three categories:

  1. Backend-only tasks: server-side integration work like data migrations, API version changes, and backend API updates.
  2. Full-stack tasks: integrations spanning server and client, requiring browser use for final submission.
  3. Gym problem sets: exercises focused on one Stripe feature area, such as Checkout or subscriptions, covering advanced configurations like shipping rates and free trial behavior.

Models outperform expectations

The benchmark deliberately biased toward difficulty, with fewer and harder tasks intended to stump the models. Based on mid-2025 evaluations of state-of-the-art models, the team expected reasonable performance on backend-only tasks and anticipated struggles with full-stack work.

The results defied those expectations. Models navigated UIs, debugged live issues, and handled underdocumented behavior. Claude Opus 4.5 was notably strong on full-stack integration tasks, averaging 92% across 4 tasks; OpenAI's GPT-5.2 excelled on gym problem sets with a 73% average across 2 tasks. All benchmarked models worked productively for long stretches, with the best runs averaging 63 turns.

In one full-stack task, agents had to upgrade an integration from a legacy Card Element UI to Checkout, then self-verify by completing a test purchase in the browser. Although no payment method was specified, one agent used Link, Stripe's digital wallet, to complete checkout end to end.

Another standout was the Checkout gym task, which required agents to reverse-engineer the API calls behind 20 prebuilt Checkout UIs. The steps involved inspecting the Checkout Session web page to identify selected products and quantities, finding product IDs in the Products API catalog, discerning session customizations like shipping amounts and custom fields, and mapping everything to the right parameters in the Checkout Session API.

An example checkout gym task. Successful solutions require both correct Checkout Session configuration and identification of the correct Products API object to pass as parameters.

Agents supplied over 80% of the correct parameters. In one complex case, a UI rendered a custom drop-down with color options—impossible to resolve from visual inspection alone. The best-performing agent clicked through the Checkout UI to discover the additional options and incorporated them as parameters.

Persistent failure modes

Ambiguity remains a weak spot. In SDK upgrade tasks, agents were asked to migrate through a breaking API version change without altering core behavior. Some agents passed nonexistent Stripe data, observed 400 errors, and declared success: "Good, the endpoint is working—it's returning a proper Stripe error for an invalid customer ID." Better runs wrote scripts to generate realistic test data and used that to validate the final submission.

Browser use also caused trouble. Agents occasionally got stuck mid-task, misunderstood browser output, or were blocked from completing actions. In one Checkout upgrade, an agent made a tool call with multiple operations that ended up highlighting the checkout UI's HTML frame and removing focus from the form. Subsequent attempts to write to input fields failed. A refresh or click outside the frame would have fixed it, but the agent concluded it couldn't recover and gave up, failing the tests.

Why benchmarks drive progress

Stripe integrations must be 100% complete and accurate. To help agents reach that bar, the company sees benchmarks as the key experimentation test bed. A replayable environment paired with a well-defined task makes it possible to prototype fixes—new prompts, skills, browser tools—and measure whether they meaningfully improve performance.

The iterative loop already showed results during the project itself. Early evaluation runs exposed how models queried Stripe documentation and MCP tools, surfacing several documentation bugs that have since been fixed.

The initial benchmark set is available in Stripe's AI toolkit, with plans to expand coverage to additional integration cases and track how model capabilities evolve. Stripe is inviting feedback and collaboration on future benchmarks.