Shopping agents are expected to handle a jacket purchase end to end — staying within budget, picking a size, reading the return policy, adding to cart. Building and evaluating such agents is harder than the task looks: catalogs, prices and layouts shift constantly, bot detection and throttling can cut runs short, and hand-made test stores cover only a narrow slice of real shopping experiences.

ShopGym is Shopify's answer to the reproducibility question: can shopping environments be repeatable without flattening the complexity of real stores? It converts live storefronts into self-contained, resettable sandbox shops and derives shopping tasks from those shops' products, features and policies. Failures can be reproduced, agents compared under identical conditions, and changes measured for actual improvement. The same environments are meant to let agents practice product discovery and policy comprehension before they touch production stores, across categories from apparel and digital goods to home improvement.

ShopGym

Two frameworks, one workflow

ShopGym splits the problem in half. ShopArena is the simulation layer, turning live seed storefronts into sandbox shops via anonymized shop specifications and a staged, validated generation pipeline. ShopGuru sits on top and synthesizes benchmark tasks in seven skill categories, each grounded in the target shop's catalog, navigation, policies and interaction affordances. Both are implemented as small sets of coding agents that communicate through the file system and run execution–verification loops to keep long-horizon generation reliable.

High-level ShopGym architecture

ShopArena, part one: exploring a live store

Exploration and generation are kept strictly apart, and a shop specification is the only thing that crosses between them. The process starts from a storefront's key surfaces — homepage, sitemap, search and cart endpoints, public catalog and policy pages — which a planner agent splits into focused tasks. Each task goes to a fresh specification agent equipped with browser tools; it explores the seed store, records evidence and writes an anonymized fragment.

The fragments together cover three dimensions of the shop:

  1. a design specification of the shop and its visual style;
  2. structured attributes for filters, navigation and pagination;
  3. catalog statistics such as product counts and price distributions.

Anonymization is built into writing: agents leave out brand names, product names, people's names and absolute URLs, so downstream stages are anonymous without a dedicated redaction pass. A consolidation step merges the fragments into a single specification, and specifications from multiple seed shops can be combined so that one sandbox carries more structural and behavioral variety than any of its sources. Since the result is human-readable, benchmark authors can inspect or edit an environment without going back to the original storefront.

Planning agent

ShopArena, part two: generating the sandbox

Generation reads the specification and nothing else — never the material gathered during exploration. From the design details and catalog statistics it invents a fictional catalog: collections, product names, descriptions, images. The storefront is then assembled in a fixed order — site shell and homepage, collections, product pages, cart, search and filtering, policy pages, and a final integration pass — so each step rests on stable output from the previous ones and a failing feature can be repaired without rebuilding the shop.

Each step repeats one loop:

  1. A fresh execution agent reads the current code, the relevant specification, the catalog and the latest feedback.
  2. It edits the code and runs lightweight checks, including type checks and builds.
  3. Rule-based checks verify the build, required routes and output structure.
  4. A visual verification agent hosts the shop, inspects it in a browser and writes feedback for the next iteration.

Fresh agents keep working context small: code carries the current state, the specification carries intent, and the latest report describes what to fix next.

Shop specifications

ShopGuru: turning shops into benchmarks

Tasks are built from what a sandbox actually contains — its products, navigation, filters and policies. Simple single-skill tasks (find this product, locate the return policy) come from rule-based generators; multi-step journeys, such as opening a collection, filtering by color, sorting by price and adding a matching item to the cart, are written by an LLM. Every referenced product, collection and filter option is checked to exist in the shop, and a task that fails goes back to the LLM with feedback for revision.

Shop data

Does the imitation hold up?

Validation ran on two tracks — structural comparison of real versus synthetic shops, then behavioral comparison of agents in both — over 224 tasks in six sandbox shops, three built from synthetic data and three twins using real data. Synthetic shops retain key structural properties of the live storefronts, and agent performance on them correlates positively with performance on the stores they mirror.

[BLOCK_2]]

Structure and behavior

Mapping pages and interactive states — open menus, cart drawers — into graphs gives the structural view. Synthetic shops have a similar count of distinct states and comparable page complexity, but fewer connections between states; the gap comes largely from external links, marketing pages and other auxiliary paths that live stores carry and sandboxes omit.

Three visualizations

Behavior was measured with two evaluation systems: BrowserGym with accessibility-tree observations, and an internal system that also uses screenshots. Success rates on live stores and their twins come out similar, suggesting the twins preserve useful difficulty. The alignment is strongest on multi-step tasks, where both systems also keep the model ranking intact. Focused tasks show larger gaps in some cases, especially in the internal system, so live-store difficulty is not reproduced exactly.

Generated sandboxes — spanning clothing, cookware, electronics and more — can be browsed on the ShopGym project site.

What comes next

Reproducibility, in this design, does not require giving up realism; the sandboxes keep meaningful signals from the live web while offering a stable place for agents to fail and improve. Planned work includes generating more sandbox shops across a wider range of retail domains, more tasks per shop, and new task types covering more of the shopping journey. The published study evaluated the quality of stores and tasks; using these environments to train agents is ongoing. The longer-term aim is to move ShopGym from a simulation and benchmarking framework to a platform for building more capable shopping agents for merchants and buyers.