A learning loop that turns failures into weights

Shopify's GraphQL agent, which answers merchant questions by writing and running queries against the Admin GraphQL API, processes up to 2,000 requests per minute in production. The system behind it demonstrates a continual learning loop, or "flywheel," that compresses production experience into the continuous space of model weights. That loop delivers higher quality than frontier models while reducing latency and cutting serving costs by 96%.

Frontier models are practical for launching a new AI product quickly, but as usage grows, they become too slow and expensive to serve every request at scale. They are also general-purpose and frozen: they do not internalize production failures, user corrections, or rejected outputs. Improvements accumulate only in discrete artifacts like prompts, retrieval examples, routing rules, and harness code. The flywheel closes that gap by continuously feeding production knowledge back into the model itself.

The loop starts with the most critical and often rushed step: defining quality. Quality is formalized as a rubric that turns product requirements into scored criteria such as completeness, execution, response quality, and safety, each with concrete anchors for what every score means. The rubric becomes the quality contract for everything downstream and the reward signal for learning.

Ground truth should include randomly sampled traffic, not only curated cases. Golden sets are useful for testing known cases, but random samples reveal what good and bad actually look like in production. Before scaling annotation, two expert annotators should blindly score about 25 random samples, and their inter-annotator agreement should be measured with Cohen's kappa. A very low score, around 0.2, means the rubric is ambiguous and needs iteration. That agreement level is the judge's ceiling: the goal is not a "perfect" judge, but one that matches humans about as well as humans match each other.

Annotations must include detail. A score with one sentence is insufficient for calibration algorithms; the reasoning behind every score is what enables learning.

Calibrating the judge and improving the baseline

The rubric is the judge's initial prompt, but it has not learned from ground truth yet. Calibration turns it into a judge that can run on infinite production datapoints. Shopify uses DSPy extensively for this, with reflection-based optimizers like GEPA and Agentic Context Engineering. GEPA evolves the prompt by reflecting on natural-language failure traces and keeps a Pareto frontier of candidates rather than greedily choosing a single winner. ACE builds a structured playbook through incremental edits.

The calibrated judge becomes the offline metric, but it is only a proxy. It must be validated against real traffic and aligned with online metrics. Shopify backtests it against prior A/B tests to confirm it recovers the direction of known wins and losses in engagement, retention, or the target behavior. Then come targeted degradation tests: deliberately making one behavior worse should cause the corresponding criterion to drop. Keeping each judge small and focused, rather than cramming all product behavior into one, makes these tests easier to interpret.

With a reliable judge, the next stage is improving the frontier-powered baseline system without touching weights. This is an autoresearch problem, in the spirit of Karpathy's recent project: an agent proposes a change to a prompt, tool definition, or harness code; evaluates it against the judge; keeps improvements; and discards failures. The entire configuration—data sources, editable directories, judge as metric, and optimizer—is defined in one markdown file, and the propose-evaluate-keep-or-discard loop runs automatically.

Training in parameter space

Once harness improvements plateau, the loop moves to parameter space. Anonymized production traffic is mined for hard negatives: conversations the judge scores low and that expose the model's weakest areas. Across millions of diverse merchants, real traffic produces a steady stream of difficult cases, from partial context to business-specific workflows and tool failures. Each failure is a bug report or Slack thread in traditional workflows; in the flywheel, it enters a self-healing pipeline.

A panel of frontier reasoning models critiques each failure, and an arbiter merges the critiques into a single repair instruction, injected before the user's turn—a technique sometimes called "hinting." The conversation is replayed and judged again. If the repair passes, the replay becomes a trajectory for reinforcement learning, with the judge's score as reward. If it fails, the case goes to human annotation, handled by Toloka's expert annotators, who score against the same rubric used to calibrate the judge.

Training runs in two stages. First, healed trajectories are distilled into a smaller model via supervised fine-tuning, training on complete trajectories—including the reasoning that produced them, a chain-of-thought distillation that lets the smaller model inherit behavior it could not learn from final answers alone. Second, GRPO is applied with the calibrated judge as the reward signal: the model samples a group of responses, judges score them, and GRPO reinforces the best performers. SFT teaches imitation of successful trajectories; GRPO optimizes directly against the definition of quality.

The self-healing pipeline runs daily, continually feeding new trajectories into the corpus. On the same cadence, a full-parameter fine-tune runs over the accumulated data, followed by GRPO. Training on new and prior trajectories limits drift and catastrophic forgetting across cycles. As the flywheel turns, quality rises and eventually surpasses the frontier-powered baseline.

Gisting to cut serving costs

A better model still has to run, and an agent's system prompt is long and static. Attention scales with sequence length, so every token attends over the entire prefix—a fixed tax on latency and serving cost for every request. Gist compression removes most of that tax. The same model runs in two ways: a teacher with the full prompt, and a student with a short sequence of learned gist tokens instead. The gist token embeddings are trained to match the teacher's output distribution while model weights remain frozen. The result is a handful of tokens reproducing the prompt's behavior with no measured quality loss.

The rewards are substantial in the GraphQL agent:

  • Higher quality. The self-healing pipeline turns low-scoring production conversations into successful trajectories, giving the model a steady stream of lessons from real merchant needs. Combined SFT and RL let the specialized model surpass frontier model performance.
  • 96% lower serving cost. Serving GraphQL traffic on a frontier model could cost an estimated $27M per year at average token costs. The fine-tuned model comes in closer to $1M—a difference between a feature painful to run at Shopify's scale and one that comfortably stays on for every merchant.
  • Lower latency. Gisting compressed the long, static system prompt from roughly 6,000 tokens to about 1,500 learned tokens. In a load test at 350 requests per minute, time-to-first-token dropped 19% and end-to-end latency dropped 38%.
  • Higher throughput. The same compression raises throughput about 16% in requests per second and 12% in output tokens per second on identical GPUs, working out to roughly 14% fewer GPUs for the same traffic.

The compounding loop

Frontier models are the starting point, but that launch advantage only takes you so far. The first round of improvements typically lands in the artifacts that wrap the model — prompts, context, tool definitions, and control flow. These changes strengthen the harness, but the model itself stays frozen.

Continual learning pushes past that boundary. The goal is to translate lessons from the harness back into the model's continuous parameter space, so that each cycle starts from a more capable model rather than just a more elaborate scaffold. That distinction matters: it is the difference between accumulating workarounds and building compounding capability.

This is how a smaller model becomes faster, cheaper, and more accurate on a specific task than the frontier baseline. The durable advantage is not any single prompt or weight update — it is the loop itself, turning production experience into progressively better weights.

References

  1. Agrawal et al. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. arXiv:2507.19457
  2. Zhang et al. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. arXiv:2510.04618
  3. Hsieh et al. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. arXiv:2305.02301
  4. Shuttleworth et al. LoRA vs Full Fine-tuning: An Illusion of Equivalence. arXiv:2410.21228
  5. Wingate et al. Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models. arXiv:2210.03162
  6. Mu et al. Learning to Compress Prompts with Gist Tokens. arXiv:2304.08467