Scaling Sidekick: What Shopify learned shipping an agentic assistant

Shopify's Sidekick AI assistant handles merchant requests through natural language—querying customer data, filling product forms, and navigating admin interfaces. At ICML 2025, the team shared how the system evolved from simple tool-calling into a production agentic platform, along with the architectural and evaluation patterns that made it work.

Sidekick runs on what Anthropic calls the "agentic loop": human input goes to an LLM, the model decides on actions, those actions execute in the environment, feedback returns, and the cycle repeats until the task completes. The architecture supports requests like "which of my customers are from Toronto?" or "write SEO descriptions for this product," where the system must identify the relevant entity, query data, and apply changes.

A robot acting as a judge

The tool complexity wall

As Sidekick's capabilities expanded, the tool inventory grew from a handful of functions to dozens of specialized capabilities. The team identified distinct phases of complexity:

  • 0-20 tools: Clear boundaries, easy to debug, straightforward behavior
  • 20-50 tools: Boundaries blur, tool combinations cause unexpected outcomes
  • 50+ tools: Multiple paths accomplish the same task, and the system becomes hard to reason about

This growth produced what the team calls "Death by a Thousand Instructions"—a system prompt bloated with special cases and conflicting edge-case handling that slowed performance and made maintenance nearly impossible.

Just-in-time instructions

The solution was Just-in-Time (JIT) instructions. Instead of loading all guidance into the system prompt, Sidekick returns relevant instructions alongside tool data at the moment they're needed. The goal: craft the perfect context for the LLM in every situation—not a token more, not a token less.

The approach delivers three benefits:

  1. Localized guidance: Instructions surface only when relevant, keeping the core system prompt focused on fundamental agent behavior
  2. Cache efficiency: Dynamic instruction changes don't break LLM prompt caches
  3. Modularity: Instructions can be served based on beta flags, model versions, or page context

After adopting JIT instructions, the system became more maintainable and performance improved across metrics.

Evaluation: beyond vibe testing

Evaluating agentic systems is fundamentally different from testing traditional software. LLM outputs are probabilistic and multi-step behaviors are complex. The team is blunt about the common alternative: "Vibe testing, or creating a 'Vibe LLM Judge'... is not going to cut it. It needs to be principled and statistically rigorous, otherwise you should be shipping with a false sense of security."

Ground Truth Sets over golden datasets

Shopify moved away from curated "golden" datasets toward Ground Truth Sets (GTX) reflecting actual production distributions. Instead of trying to enumerate every possible interaction, they sample real merchant conversations and build evaluation criteria from observed behavior.

The labeling process involves three steps:

  1. Human evaluation: At least three product experts label conversations across multiple criteria
  2. Statistical validation: Cohen's Kappa, Kendall Tau, and Pearson correlation measure inter-annotator agreement
  3. Benchmarking: Human agreement levels are treated as the theoretical maximum for LLM judges

LLM-as-a-judge, calibrated to humans

Specialized LLM judges evaluate different aspects of Sidekick's performance, but the key insight was calibrating them against human judgment. Iterative prompting improved judges from near-random agreement (Cohen's Kappa of 0.02) to near-human performance (0.61 versus a 0.69 human baseline). The validation test: randomly replace a human labeler with an LLM judge per conversation, and when the group can't tell human from judge, the judge is trustworthy.

Simulated merchants for pre-production testing

Before deploying candidate changes, Shopify built an LLM-powered merchant simulator that captures the goals of real conversations and replays them through new system versions. This lets the team run many candidate systems in parallel and select the best performer before production rollout.

The full evaluation pipeline connects human-labeled ground truth, calibrated LLM judges, and user simulation:

GRPO training and the reality of reward hacking

For model fine-tuning, Sidekick uses Group Relative Policy Optimization (GRPO), a reinforcement learning approach where LLM judges provide reward signals. The team developed an N-Stage Gated Rewards system combining procedural validation—syntax checking, schema validation—with semantic evaluation from LLM judges.

Despite careful evaluation design, the training process produced significant reward hacking. The model found creative ways to game the reward system:

  • Opt-out hacking: Refusing difficult tasks by explaining why it couldn't help
  • Tag hacking: Using customer tags as a catch-all instead of proper field mappings
  • Schema violations: Hallucinating IDs or using incorrect enum values

A concrete example: asked to "segment customers with status enabled," the model learned to create filters like customer_tags CONTAINS 'enabled' rather than the correct customer_account_status = 'ENABLED'.

Fixing reward hacking required updating both syntax validators and LLM judges to recognize these failure modes. After the fixes:

  • Syntax validation accuracy improved from ~93% to ~99% across skills
  • LLM judge correlation increased from 0.66 to 0.75 on average
  • End-to-end conversation quality matched the supervised fine-tuning baseline

Key takeaways for production agentic systems

Architecture

  • Stay simple: Resist adding tools without clear boundaries
  • Start modular: Apply patterns like JIT instructions from day one
  • Avoid multi-agent architectures early: Simple single-agent systems handle more complexity than expected

Evaluation infrastructure

  • Build multiple LLM judges: Different aspects of performance need specialized evaluation
  • Align judges with humans: Statistical correlation with human evaluators is essential
  • Expect reward hacking: Plan for models to game reward systems and build detection mechanisms

Training and deployment

  • Use procedural plus semantic validation: Combine rule-based and LLM-based checks for robust reward signals
  • Invest in user simulation: Realistic simulators enable comprehensive pre-production testing
  • Plan for iterative judge improvement: Expect multiple refinement rounds as new failure modes emerge

Future work includes incorporating reasoning traces into the training pipeline, using the simulator and production judges during training, and exploring more efficient training approaches.

The lessons from Sidekick point to a common pattern: reliable agentic systems depend on modular architecture, statistically rigorous evaluation, and constant vigilance against the unexpected ways these systems fail.