Synthetic shoppers for the long tail of A/B testing

A/B testing a storefront is a luxury. It requires traffic, and lots of it. Merchants without that traffic — typically smaller operations — are left to guess whether a new theme or layout actually converts better. Shopify's SimGym product attempts to solve that problem by replacing real visitors with synthetic ones: fleets of LLM-driven robots that browse a store in cloud browsers, each with a persona, budget, and shopping intent, acting on what they see rather than against a mocked DOM. A twin robot runs the same scenario against the alternate theme, producing a simulated A/B test in minutes instead of weeks.

That simulation loop is expensive to run at scale. Each robot session involves a cloud browser from Browserbase — up to 2,000 concurrent Chromium instances — tightly coupled to Shopify's inference cluster. Browserbase streams the page state to the model, which returns a JSON action that Browserbase executes. This repeats 10–13 times per session, with the model consuming a one-sentence system prompt, a buyer profile, accumulated memory, and a representation tree of the current page (3,000 to 15,000 tokens).

The commercial model creates a hard operational constraint: Browserbase bills per minute, while inference takes seconds. Latency is not just a performance metric; it is the dominant cost driver. Input tokens make up 94% of all tokens consumed, and a ~20% reduction in average LLM latency was measured to cut cost per merchant run by ~10% and increase daily throughput by ~12%.

From API calls to self-hosting

The first iteration of SimGym used GPT-4o and other models via standard APIs. It worked — 75 of 100 simulated buyers completed their tasks — but at hundreds of thousands of sessions per day, per-token API costs were unsustainable. The pivot was to gpt-oss-120b, an open-source 120B Mixture-of-Experts model, combined with the Notte browser automation framework. Session times dropped from over 15 minutes to under 3, and costs became predictable. Self-hosting also opened the door to task-specific fine-tuning.

That decision created the next bottleneck: hosting. Groq's hosted API was fast but capped at 5 million tokens per minute with 50 concurrent sessions. A move to NVIDIA/CentML removed rate limits but exposed 10x latency spikes under high concurrency. The workload clearly required dedicated, custom infrastructure.

Why agentic traffic breaks standard serving

SimGym's traffic pattern is the opposite of a typical chatbot workload. Instead of independent, short, randomly arriving requests, SimGym fires 600 correlated long-context requests at once. Each request depends on the previous one, requires JSON schema enforcement, and accumulates 89K–127K tokens over a session. Every step is strictly sequential, so there is no parallelization across steps — the only way to speed up a session is to speed up each individual step.

The model itself adds another wrinkle. gpt-oss-120b is a MoE model: only a fraction of its parameters activates per token, so compute per token is comparable to a 15–20B dense model. But all 120B parameters must reside in GPU memory, making the workload memory-bound. The bottleneck is loading expert weights from HBM, not arithmetic, and standard vLLM serving was not efficient at that.

Optimizing for Blackwell

Shopify already had a working relationship with CentML (acquired by NVIDIA eight months prior) on inference optimization for Search, Product Classification, Recommendation Systems, and Sidekick. SimGym required its own track, with the application team, inference platform, and GPU kernel engineers collaborating on profiling data and production traffic shapes.

The first task was getting gpt-oss-120b onto Blackwell B200 hardware. Since the model is memory-bound, the biggest wins came from reducing HBM traffic: MXFP4 quantization for expert weights, FP8 attention for the KV-cache, and optimized FlashInfer kernels for attention and MoE operations. The team also tuned chunked prefill and request scheduling, and fixed bugs that surface only under production agentic load (vLLM PR #28000). The baseline result: 80K tokens per second on B200.

Standard benchmarks weren't actionable for this workload, so NVIDIA collected production traffic shapes, prompt distributions, concurrency patterns, and token lengths, then replayed them against H100, H200, and B200 hardware. The replay showed 11K tokens/second per H200 versus 57K tokens/second per B200 — a 5.2x speedup, making Blackwell the clear choice.

Custom kernels and integration work

The workload shaped several engineering efforts. NVIDIA built custom FlashInfer kernels for long-context speculative decoding on Blackwell, based on profiling data from Shopify's workload. These went upstream as FlashInfer PR #2265 (available in FlashInfer 0.6.1) and delivered roughly a 2x attention kernel speedup for speculative decoding at long context.

SimGym also needed three features to work simultaneously: async scheduling for throughput, guided decoding for JSON schema enforcement, and speculative decoding for latency. Making them interoperate without sacrificing individual gains led to a vLLM contribution (PR #29821).

Speculative decoding matters here because the output is structured JSON. Field names like "action", "nodeSelector", and "method" are highly predictable, so a small draft model's proposals are accepted at a high rate. NVIDIA trained and published an EAGLE-3 speculative decoding head for this purpose: nvidia/gpt-oss-120b-Eagle3-throughput on HuggingFace.

Runtime work already in production — async scheduling, stream interval buffering, and torch.compile kernel fusion — contributed immediate gains. The vLLM and NVIDIA teams published full details in a joint blog post, reporting a 38% throughput increase and 13% latency improvement overall, with teams from Red Hat, NVIDIA, vLLM, and Meta involved. Production results so far show a 10% speedup from async scheduling (80K to 88K TPS per B200) and a 57% reduction in HTTP/gRPC overhead from stream interval optimization. In benchmarks, speculative decoding added a further 6% at 100–200 concurrent sessions (33K to 35K TPS) and is next to go live.

Today's stack and the MIG alternative

The current production setup is 48 dedicated NVIDIA B200 GPUs on CentML with a fixed allocation and no autoscaling. The serving stack is vLLM wrapped by CentML's cserve acceleration layer (with a custom build for gpt-oss), with FlashInfer providing the GPU kernels. Speculative decoding via the EAGLE-3 head is validated and queued for deployment.

The team also investigated MIG (Multi-Instance GPU) partitioning, which splits each B200 into two isolated instances. In end-to-end experiments at production QPS, MIG reduced average LLM latency by ~20% (27.8s to 21.9s), dropped session duration from 7.3 to 6.6 minutes, and increased daily throughput from 1,311 to 1,463 merchant runs, with near-linear scaling and no quality regression. MIG has a different optimization profile than speculative decoding: partitioning GPU memory means the draft model doesn't fit alongside the main model, and prefill-decode disaggregation doesn't apply. But the gains from doubling serving instances are substantial on their own.

Prompt restructuring and a dead end

Prompt changes also helped. Restructuring the system prompt to move dynamic elements (persona, intent) out of the shared prefix improved prefix cache hit rates. With caching enabled, NVIDIA's experiments showed ~12% throughput improvement at concurrency above 1,000, plus meaningful time-to-first-token gains at all concurrency levels.

Lowering reasoning effort was tried and rejected: session duration dropped ~75%, but error rates jumped from 0.5–0.75% to 4.5–10.9% across trials.

What's next

The collaboration produced a set of validated optimizations, and the question now is sequencing. MIG partitioning is closest to production because it doubles serving instances and gives ~20% latency reduction and ~12% more daily throughput with no quality regression.

On full-GPU configurations, a different set of levers opens up. Speculative decoding with the EAGLE-3 head is validated and ready. Disaggregated serving — separating prefill and decode so they scale independently — is the next major architectural shift, particularly important given that 94% of tokens are input. Additional FlashInfer improvements, including RoPE+Q+Cache fusions, are also on the roadmap. Shopify is also pursuing prefix caching in production and smaller finetuned models trained on 16 H200 GPUs on Nebius, with Qwen3-32B already a candidate.

Seven months after starting as a prototype making API calls, SimGym now runs on a Blackwell cluster with custom kernels and a pipeline of optimizations that spawns 400,000 shopping sessions a day. Cost per merchant run is in single digits and falling. A merchant who never had enough traffic for an A/B test can now get results in four minutes — backed by a crowd of robots.

Building a better experiment, and finding a hiring partner in the process

Running thousands of simulated customers through a virtual storefront is how Shopify’s engineering team de-risks the A/B tests that shape the platform. The simulation system, which walks bots through a model of the checkout experience, lets developers compare variants without the noise and cost of production traffic. It also turned out to be a reliable way to spot good engineers — the kind who volunteer to spend weeks debugging a simulator rather than shipping another dashboard.

From live tests to laboratory conditions

Real-world A/B testing has an inconvenient dependency on real-world users. If a proposed change to the checkout flow might reduce conversion, the only way to know is to expose a fraction of merchants’ customers to it, wait for statistical significance, and hope the weekend traffic spike doesn’t confound the results. That process is slow, expensive, and occasionally unfair to the merchants whose buyers get the worse variant.

The team wanted a controlled environment where they could replay historical sessions, inject edge cases, and test a hundred variations in the time it takes to run one live experiment. The result is a simulation engine that models a shopper’s journey from landing page to payment confirmation. Each simulated agent follows probabilistic rules derived from observed behavior: some browse quickly, others hesitate at shipping costs, a few abandon carts at the last step. By tuning those rules against known outcomes, the team can validate that the simulator reflects reality.

What the simulator actually does

At its core, the system is a batch job. It takes a set of experiment variants, spins up a headless browser for each, and drives thousands of scripted interactions through the storefront. The agents are not simple random walkers; they are governed by a behavioral model that encodes the likelihood of clicking a product image, adding an item to the cart, entering a discount code, or navigating to a different category. The team calibrates these probabilities against anonymized production data, then uses the simulator to forecast how a change would shift the overall funnel.

  • Variant comparison: run the same agent population against the control and each treatment to measure the delta in completion rates.
  • Edge-case injection: force rare paths—out-of-stock items, expired gift cards, geo-blocked shipping—that might take weeks to occur organically.
  • Regression hunting: replay a known-good session against a new build to catch rendering or logic breaks before any human sees it.

The simulator does not replace live experiments. It complements them: cheap, fast, and exhaustive, while production tests remain the final arbiter of customer impact.

The team behind the bots

The engineers who built this are the same people who debug a GPU kernel one afternoon and tune a fine-tuning pipeline the next. They run production services and write conference papers with equal comfort. The project attracted them precisely because it was hard and unglamorous — the kind of work that requires patience with flaky test suites and an instinct for when a simulation has drifted from reality.

That temperament matters more than the credentials on a résumé. The hiring bar for this team is not about years of experience or a particular degree, but about demonstrated curiosity and a willingness to dig into messy problems. If the idea of herding 2,000 simulated shoppers through a virtual checkout sounds like a fun Tuesday, the team is hiring. Join them.