Sidekick, Shopify's AI commerce assistant, uses a frontier model for the conversation with the merchant, planning, and tool calls. Focused jobs get handed to subagents, and several of those run open models that Shopify has post-trained for a single task. The GraphQL subagent is the clearest example: it reads Shopify's API schema, searches docs, writes and executes a query, and returns results. The workload is context-heavy — roughly 40 tokens read for every token written — and a frontier model's API bills for all of them.
For that subagent, Shopify post-trained NVIDIA Nemotron 3.5 Lightning. On the task it scores slightly above the frontier model, answers 7× faster, and costs about 90% less to serve. Owning the post-trained weights also means Shopify can run the model on its own GPUs and control when it changes.
The optimization flywheel
Task experts are built with Sidekick's continual learning loop. Three setup steps define the target, then the loop repeats with each new batch of production data.
Define a scoring rubric
Product experts turn requirements into a rubric; for GraphQL it covers completeness, execution, and response quality. Inter-annotator agreement sets the bar, and an LLM judge is tuned with the GEPA and ACE optimizers until it matches held-out human labels equally well. After that, the judge scores every agent change, candidate model, and training run.
Push the frontier agent first
The frontier-model version of the agent is improved before any open model is considered. An autoresearch loop proposes one small change at a time to prompts, tool descriptions, or control flow, keeping it only when the judge's score rises. That improved agent is the bar the open model must beat.
Pick and evaluate an open model
Shopify does not pretrain. Candidate open models such as NVIDIA Nemotron are tested inside the real subagent with the same tools and judge, weighing starting quality, serving speed, and support in NVIDIA Megatron Bridge, the training stack in use. GraphQL landed on Nemotron 3.5 Lightning, a mixture-of-experts model with three billion of its 30 billion parameters active per token.
Measure inference early
Serving is measured from the outset, since a model too slow for production isn't worth post-training. Workloads are benchmarked with NVIDIA AIPerf, an open-source load tester in NVIDIA Dynamo that replays production traces with synthetic text instead of customer data. Shopify contributed live-response replay support, so multi-turn tests hit the server's prefix cache the way production does. Results steer the serving stack: for Buyer Profile, a batch job, AIPerf compared vLLM, SGLang, and NVIDIA TensorRT-LLM, and TensorRT-LLM did the most work per GPU. In NVIDIA's measurements, the SimGym shopping simulator ran 4.5× faster on Blackwell B200 than on H200 at 1.48× lower cost.
Harvesting failures as training data
Each night the judge scores a sample of production GraphQL conversations, and anything below perfect goes to self-healing. A frontier model acting as critic writes a hint — never the answer — and a simulated merchant reruns the request with it against the same store. If the judge passes the rerun, the hint is removed, the conversation anonymized, and the example added to the training set; otherwise the critic retries, up to three times.
Over 86 nightly runs, self-healing repaired 59,535 GraphQL conversations, or 69% of the failures it attempted. Human annotators corrected the remainder, and those corrections feed the ground truth used to calibrate the judge.
Training and results
Post-training runs on NVIDIA Megatron Bridge, which brings standard Hugging Face checkpoints onto NVIDIA's Megatron Core library without rewriting the model and scales across nodes on Nebius through settings rather than code. On the same FP8 post-training job for the Analytics model, eight B300 GPUs trained 1.86× faster than eight H200s and used 46% fewer GPU-hours. The judge then scores the new checkpoint and the loop begins again.
On the GraphQL judge, Nemotron 3.5 Lightning started at 37.7 out of the box against 73 for the frontier model. The first loop raised it to 61.5, and after six loops and 61,000 healed examples it reached 74.5.
Replaying production GraphQL traffic with AIPerf, the post-trained model returned a full response in 0.87 seconds at the median versus 6.50 seconds for the frontier model's API, and its slowest 1% of requests finished 8× sooner. It also beat another post-trained open model of comparable size, which was close at the median but more than twice as slow at p99. Serving cost on that traffic is about 90% lower than the frontier model's, and across Shopify's post-trained models self-hosting cuts serving cost by 90–98% against frontier pricing.
Division of labor
The frontier model keeps what it does best: talking to the merchant and deciding which tools to call. Token-heavy specialist work moves to open models Shopify owns, and each turn of the flywheel compounds — human corrections sharpen the judge, production traffic grows the training set, and each NVIDIA hardware and software generation makes the next loop faster and cheaper to run.



