Agentic Testing: Where Agents Fit in the E2E Testing Stack

Goals Versus Journeys: What Agentic Testing Actually Changes

Traditional end-to-end tests validate a specific journey through the UI: click → click → type → assert. Agent-driven tests validate whether a goal can be achieved, typically expressed as an instruction such as "send a thread message." The difference is fundamental: tests enforce journeys, while agents verify goals.

In our runs, the overall workflow remained consistent (login → search → result → clear), but the exact sequence of actions varied. Agents reached the same outcomes through different input methods (clicking a search suggestion vs. pressing Enter), different navigation patterns (reopening search vs. reusing existing state), and with additional or skipped steps. This flexibility is useful, but it carries tradeoffs in reliability, cost, and execution time.

The immediate question is whether something costing $15–30 per run and taking over 10 minutes can fit into modern testing workflows. Based on 200+ runs, we found that agent-driven tests are fundamentally different from traditional tests — not a replacement for them — and they can be highly reliable with a clear place in the stack.

Experiment Design

We evaluated three execution models across a matrix of configurations, each run 20 times:

• Agent + Playwright MCP: The agent interacts with the browser through predefined actions with persistent context (DOM snapshots and logs).
• Agent + Playwright CLI: The agent runs Playwright CLI commands via the shell, executing one step at a time and deciding the next action based on updated UI state.
• Generated Playwright Tests: An AI agent generates deterministic test code from a natural language description, executes it as a standard E2E test, and iteratively refines it until it passes.

We used two flows: Thread Reply (simple, ~15–20 steps) and Search Discovery (medium, ~25–30 steps). For agent-driven approaches, we evaluated both natural language instructions and structured YAML with explicit steps, actions, and expected outcomes. All experiments ran in test workspaces using non-production data.

Agent model for MCP/CLI runs: Claude Sonnet 4.5. Model for generated tests: Claude Opus 4.6. Execution was non-interactive via Claude Code (claude -p).

Reliability: The Gap Widens With Complexity

Playwright MCP was the most reliable agentic configuration, achieving near-zero failure rates on simple scenarios and staying within 0–12% on complex flows. Playwright CLI showed higher failure rates (roughly 12–20%), with most failures caused by execution issues — authentication handling, navigation timing, and session instability — rather than model reasoning.

Generated Playwright tests performed reasonably on simple flows (~8% failure) but degraded significantly on complex workflows (~48%). These tests typically progressed through 70–80% of the flow before breaking on a final interaction or assertion. Failures stemmed from variability in UI state and abstraction mismatches, particularly when reused page objects interfered with precise element targeting.

The reliability gap likely comes down to state handling. MCP keeps a live, stable view of the app, while CLI rebuilds state from snapshots at each step. As flows get longer, small inconsistencies in UI interpretation or timing accumulate into failures. Additionally, MCP-based agents appear to reuse successful interactions from earlier steps in the same flow, while CLI-based execution feels more like starting from scratch each time.

Speed: Generated Tests Win on Repetition

Generated tests were consistently the fastest, even including generation time in the measurements.

For generated tests, raw execution was much faster than the averaged numbers suggest: ~32 seconds for Thread Reply and ~45 seconds for Search Discovery, with generation happening once and execution repeated five times. In CI environments, the one-time generation cost becomes negligible, allowing deterministic tests to scale efficiently.

Agent-driven workflows pay the observation-reasoning-execution cost on every run and every step.

Adaptability: Paths Differ, Goals Stay Constant

Only about 20% of agentic runs followed the exact same action sequence. In most runs, agents discovered different valid UI paths — opening menus in different orders, selecting slightly different elements, or using alternate navigation flows — while still reaching the correct final state.

To measure this, we compared normalized action signatures (ordered lists of tool calls and UI actions) across runs, collapsing parameters, wait/snapshot actions, and equivalent tool variants. Most sequences differed even when outcomes were correct, reinforcing the core distinction: deterministic tests enforce one journey; agents explore and verify that the goal state is reachable.

Cost: Context Retransmission Dominates

Agent-driven runs typically cost $15–30 per execution. Token analysis on the same Search Discovery flow showed that the execution model mattered more than the underlying AI model.

The API underneath Claude Code is stateless: every turn re-sends the full system prompt plus all prior conversation history. Cost is therefore driven by turn count and context growth, not model output.

CLI took an average of 85 turns versus MCP's ~40–60 because each browser interaction was split across multiple commands (actions, waits, snapshots, reads, element lookups), while MCP combined interaction and state return into a single round trip. Each additional turn pays the full system prompt tax plus re-sends all prior context.

The context payload itself consists mainly of browser snapshots (accessibility tree snapshots for MCP/CLI) or test runner output with error traces and DOM state (for generated tests). The majority of cost came from retransmitting previously seen content; only a small fraction of tokens represented new information per turn. Prompt caching, context compaction, and reduced snapshot frequency are potential optimizations we did not apply.

Infrastructure Matters as Much as the Model

The execution environment significantly affected reliability.

Most CLI failures came from authentication and navigation issues — sign-in errors, timeouts, session instability — indicating failures at the execution layer, not in reasoning. MCP's structured browser primitives and tighter integration with the agent's tool-calling workflow reduced these issues. MCP runs were also easy to execute concurrently, while CLI runs were difficult to parallelize in our setup.

Our experiments covered single-session UI workflows only. Cross-workspace flows or multi-window scenarios introduce additional complexity: MCP may face cost issues as observation loops lengthen, and CLI adds coordination overhead for multiple browser sessions on top of its higher token usage.

Where Agentic Testing Fits

Agent-driven E2E tests are not a replacement for deterministic tests. They add an exploratory layer suited for targeted debugging and exploratory testing, where their adaptability and ability to verify goals provides value that scripted journeys cannot. At current cost and speed, they are less suited for high-frequency CI execution, though improvements in models and tooling may shift that balance.

Agentic Testing’s Place in the Stack

Agent-driven testing doesn’t replace the existing testing stack—it adds a new capability on top of it. The distinction comes down to how workflows are executed rather than where they operate.

Deterministic E2E Tests

Traditional end-to-end tests remain the best fit for fast, repeatable regression checks in CI. These tests are human-written or AI-generated, enforce a specific journey through the UI, and carry a low operational cost. They are quick, deterministic, and CI-friendly.

Agentic Testing

Agent-driven workflows take a fundamentally different approach. Instead of executing a predefined script, agents work from a goal: they observe the UI, reason about the current state, and determine how to reach the desired outcome. This makes them well suited for:

• Exploring complex UI behavior
• Debugging flaky workflows
• Reproducing production bugs

From a system perspective, agentic testing still validates real user workflows through the UI at the same level as E2E tests. The difference is entirely in the execution model.

That distinction points to a combined strategy going forward. Deterministic tests give teams a stable foundation for CI, while agentic testing adds a distinct layer at the top of the testing pyramid for exploration, debugging, and validating complex behaviors. Both approaches have a role, and the most effective testing strategies will use them together.

Acknowledgements

The DevXP AI team built and supported the tools—including Claude Code and the metrics infrastructure—that made these experiments possible. That foundation made it much easier to run, analyze, and iterate on hundreds of executions.

Thanks also to managers Dave Harrington and Vani Anantha for supporting experiments at a scale that kept token counters busy, and to the Frontend Test Frameworks team for their help from early ideas through validation and feedback. Special thanks to Lucy Cheng, Natalie Stormann, Roopa Thanisraj, Ilaria Varriale, and Crescencio Zul for their thoughtful input.

If this kind of work—pushing the boundaries of testing or building agent-driven systems and rethinking developer workflows—sounds interesting, the team is hiring.