Closing the refusal gap in training data
Production training logs capture successes. By construction, they omit the requests a model should decline — impossible queries, missing capabilities, ambiguous phrasing. When Shopify's Sidekick assistant was trained only on those successful examples, it had no learned behavior for edge cases that required a refusal. Instead, it improvised, generating syntactically valid queries that silently returned zero results. Merchants saw not "this can't be done," but "no customers match."
The team hit this while building customer segmentation skills for Sidekick. The model had to translate plain-English merchant requests into Shopify's domain-specific query language. Fine-tuning on tens of thousands of de-identified production queries handled the common cases well, but every example was a success story with zero refusals. The model had never learned to say no.
Why naive data merging failed
The obvious fix was to add refusal examples to the corpus. Shopify partnered with the Toloka team to produce a balanced dataset of roughly 600 standard queries and 602 refusal annotations. Merging these with the production corpus and fine-tuning produced limited improvement.
The problem was contradictory training signals. The same query — or semantically similar requests — appeared in both datasets with opposite labels. A query that passed evaluation in production might be flagged as impossible in the Toloka data. With tens of thousands of production examples, reconciling those conflicts by hand was not feasible.
An automated curation pipeline with judge consensus
Instead of scaling up manual annotation, Shopify treated the Toloka dataset as a seed for an automated curation engine. A panel of four frontier LLMs, anonymized as A through D, ran as automated data judges across the full training corpus.
Calibration before deployment
Each judge was calibrated first, using few-shot examples from the Toloka dataset that paired representative queries with their ground-truth labels. This anchored the judges' decisions to what human annotators actually flagged as impossible, rather than to assumptions about Shopify's data schema.
Once calibrated, the ensemble did double duty. It labeled new queries and resolved conflicts when a query had different labels across sources, evaluating each case against the seed distribution to assign a final verdict.
Strict consensus over confidence scores
The pipeline gates every label through unanimous agreement. All four judges must reach the same verdict and provide consistent reasoning before a label change is accepted. Disagreements are filtered out, not arbitrated.
This prioritizes precision over recall. If four independent models can't agree, the example goes to a human. The judges did disagree often — filtering out those edge cases prevented erratic training shifts and kept precision high while the team was still mapping the problem space.
A taxonomy with no overlap
Judges classified each query into one of four mutually exclusive categories:
- Solvable with more context: the outer planner needs to fetch additional information first.
- Missing capability: the request asks for a Shopify segmentation feature that doesn't exist yet.
- Wrong skill: not a segmentation task; the planner should route to analytics instead.
- Ambiguous: needs clarification from the merchant.
Mutual exclusivity mattered. When categories overlap, judges disagree and labels become inconsistent — inconsistencies that propagate downstream into unstable fine-tuning.
The flywheel: production traffic as training fuel
The pipeline's real value emerged once every production gap fed the next training run. When the improved model ships, its production traffic becomes the next sampling pool. Merchants keep finding new phrasings, and partial requests expose gaps absent from the original training set.
Each cycle repeats the process: the judge ensemble labels new patterns, examples with judge disagreement are dropped, and accepted labels return to the corpus. Every fine-tuning run starts from a bigger, cleaner baseline, and each deployment funds the next improvement.
Results and validation
Enabling refusal capabilities lifted the segmentation skill evaluation score from 0.619 to 0.798, a 28.9% relative gain. That headline number needs context: the production model started with zero refusal examples, so part of the gain reflects adding any refusal data at all. Comparing strategies from the same refusal-aware baseline, automated curation pushed the pass rate from 0.762 to 0.798 over naive merging.
Manual validation showed refusal accuracy at 86.3% with a false positive rate of 4.6%. Agreement between the judge ensemble and the ground-truth seed data was strong: near 90% prediction accuracy with a Cohen's kappa above 0.75 across all four models — substantial agreement by standard interpretation.
Four lessons from building the curation pipeline
Small seed datasets have outsized leverage
The Toloka annotations drove the entire pipeline not through volume but through quality. Garbage in, garbage out applies with extra force here — judges inherit whatever biases or errors the seed contains. A small, high-quality annotated set can power an automated curation engine that no manual labeling budget could replicate.
Mutually exclusive categories prevent downstream compounding
Ambiguous category definitions create judge disagreements that multiply as labels flow into training. The upfront work of defining clean, non-overlapping categories paid for itself — calibration was faster, labels were more consistent, and fine-tuning was more stable.
Consensus beats confidence early on
When the shape of the problem is still unknown, unanimous agreement across models is worth more than any single model's confidence score. Multi-judge consensus kept precision high while edge cases were still being discovered. If four independent models can't agree, that example belongs in front of a human.
Refusals are product features, not failures
A hallucinated answer is the worst outcome for a system that can't fulfill a request. A truthful refusal — ideally with a suggestion for what to try instead — lets the outer planner keep a conversation productive. Shopify spent considerable effort teaching Sidekick when to say yes. The harder, earlier lesson was teaching it when to say no.
A Pattern, Not a Patch
The refusal-classification system was never intended as a one-off fix. It is infrastructure: every deployment produces new signal, and every gap that surfaces becomes the starting point for the next annotation round. That same loop can be applied whenever production traffic fails to capture the specific errors you need to address.
Shopify is currently extending this framework to other skill models inside Sidekick, using the same process for additional data-quality issues. The per-domain details shift—different models, thresholds, taxonomies—but the architecture holds: a small, high-quality seed set; judges calibrated against it; a unanimous consensus gate; and a feedback loop that routes learnings back into production.
What proved most demanding was not the machine learning itself. The real effort went into building the surrounding infrastructure robustly enough that the model could keep improving. That, in essence, is the work of a data flywheel.
