Teaching a model to build automations
Shopify Flow lets store owners automate workflows from triggers, conditions, and actions. But building a workflow from a blank canvas is intimidating for non-engineers. Sidekick, Shopify's AI commerce assistant, handles that by generating a Flow from plain English. The problem: closed frontier models with generic prompting are expensive, slow, and easy to copy. Any competitor with an API key can replicate them. What makes an AI feature defensible is proprietary data, the training recipe, and a tight feedback loop.
Shopify has a unique advantage: millions of merchant interactions that directly signal whether a generated workflow is any good. Those signals only help if the system actually learns from them. So Shopify fine-tuned Qwen3-32B into a tool-calling agent for Flow generation. The result is 2.2x faster and 68% cheaper than the prompt-based alternative, with higher accuracy.
The path there was not straightforward. Three problems dominated: cold-start data, a domain-specific language (DSL) that models struggle to learn, and benchmark scores that lied about real-world performance. The fixes — working backwards from production artifacts, teaching the model to write Python instead of JSON, and building a weekly retraining pipeline — are the substance of what follows.
Solving the cold-start problem
The feature hadn't been deployed, so there were no production conversations to train on. To bootstrap the dataset, Shopify reverse-engineered user intent from existing workflows. Thousands of store owners had already built Flow workflows manually. Those were sampled and filtered: workflows that ran at least once in the past seven days, from merchants with two or more qualifying workflows, with only one example per descriptor to ensure diversity.
From those validated workflows, training data was constructed in three steps:
- Sample a workflow. Pick a popular, production-validated workflow.
- Generate a user query. Use a stronger LLM to produce a plausible natural-language request that would lead to that workflow.
- Construct the tool trajectory. Build the full multi-turn sequence of tool calls an ideal agent would execute to arrive at the workflow. This was the bulk of the engineering effort.
The model was fine-tuned on this synthetic data and evaluated against a benchmark of 300 hand-crafted examples covering the breadth of expected Flow usage. An LLM evaluation framework checks semantic correctness (does the workflow do what it should?) while syntactic correctness (malformed conditions, invalid references, bad configurations) is validated programmatically. Latency — time from request to delivery — was tracked as the third metric.
Teaching the model to write Python instead of JSON
Flow workflows are represented internally in a JSON-based DSL designed for backend parsing and execution. That format works well for production systems but poorly for LLMs. Conditional, program-like logic that would normally appear as code ends up buried in deeply nested JSON — a pattern rare in pretraining data. Forcing the model to learn the native format from scratch meant asking it to learn a new language and a new task simultaneously.
Shopify's solution: reformulate the task in a representation closer to what the model already knows. Workflows are programs, so the model was taught to write them in Python. A transpiler converts the JSON DSL into semantically equivalent Python, and back again. The fine-tuned model draws on familiar patterns — decorators, if/else, variables, loops, function calls — instead of fighting an unfamiliar data format.
The difference was dramatic. With identical training data, switching from JSON DSL to Python DSL improved syntactic correctness by 22 points and semantic correctness by 13 points. The problem shifted from "learn a new language and the task" to just "learn the task."
Building that round-trip transpiler required handling the full complexity of Flow logic without losing meaning in either direction. Reliability came from extensive testing: every workflow merchants created through Sidekick in production was round-trip tested, converted from JSON to Python and back, with the output verified to match exactly. Any mismatch was caught before it could contaminate training data.
At inference, the model writes Python. The transpiler converts it to JSON for the Flow backend. Store owners never see Python, and the backend never needs to understand it. Python exists purely as the model's internal language.
Prior work has explored Python as an intermediate representation — SPEAC, LLMLift, WorkflowLLM — but via prompting or without a round-trip transpiler. What distinguishes this approach is the closed loop: fine-tuning on Python combined with a transpiler back to the production DSL, with no downstream system changes required.
Mirroring production exactly
Representation choice was one half of the data problem. The other was ensuring training data precisely matched production conditions. The model proved highly sensitive to the degree of match — every difference closed, however minor, improved eval scores.
- Tool naming and ordering: Training used the full prefixed name
flow_app_agent_task_search; inference called ittask_search. Functionally identical, but the model treated them as different tools. Removing the prefix from training data improved accuracy. Tool ordering in the system prompt mattered too — shuffling between training and serving degraded performance. - Tool response format: Tool responses return JSON objects with multiple fields. Training data sorted keys alphabetically. Production returned them in a different order, or with an extra field, and the model noticed. Any drift between training data and production APIs hurt accuracy.
- System prompt and tool descriptions: Tool descriptions changed frequently as the product team iterated. Every update had to be reflected in training data or behavior drifted. Keeping the two in sync was an ongoing process, not a one-time fix.
None of these affect the logic of the task. They are formatting details — but the model treats every token as a signal, whether intended or not.
Keeping context small
Tool calls add their responses to context, which grows latency and cost with every step. More importantly, irrelevant context dilutes reasoning accuracy. Shopify restructured tool interfaces to minimize context: tools return lightweight summaries first, the model scans and selects what it needs, then retrieves full details only for the selected items. Two cheap calls replace one expensive one.
Flow has hundreds of triggers, conditions, and actions. A search might return 100 matches. Instead of loading the full configuration schema for each, task_search returns just names and descriptions. The model picks the 2–3 it actually needs, then calls task_configuration for full schemas only on those. Context stays small; reasoning stays focused.
Retraining every week
As the data pipeline grew, more training data meant better accuracy but slower runs. Slower runs meant fewer iterations, which meant slower improvement. Shopify needed the full dataset and weekly retraining. The infrastructure now supports both.
Qwen3-32B trains on two nodes of H200 GPUs with Fully Sharded Data Parallel (FSDP). A full training run takes 12 hours — fast enough for weekly retraining with experimental runs in between. The pipeline, from data collection through training to evaluation and deployment, runs on Tangle, Shopify's open-source ML experimentation platform. Each step composes into a single reproducible workflow with intelligent caching: only affected steps re-run when something changes.
CometML tracks runs, HuggingFace hosts datasets and checkpoints, and CentML serves the model in production. Weekly retraining proceeds without manual intervention.
When benchmarks don't match reality
Synthetic data brought the fine-tuned model to parity on offline benchmarks. Every tracked metric said it was production-ready. It was deployed to 1% of traffic to verify.
The activation rate — whether store owners actually enabled the generated workflows — came in 35% lower than the prompt-based agent. The benchmark covered what Shopify expected merchants to ask, not what they actually asked: editing existing workflows, configuring email handlers, working with third-party integrations, and asking questions about Flow without intending to create one.
The model was strong in-domain, but real traffic exposed out-of-distribution requests the synthetic data hadn't covered. Activation rate proved noisy — it reflected merchant behavior more than model quality — so Shopify optimized for a domain-expert-calibrated LLM judge while keeping activation rate as a guardrail against regression. The lesson holds for any agentic system: offline metrics can show parity while real-world usage reveals a fundamentally different distribution.
Turning production traffic into training data
When the fine-tuned model reached 1% of production traffic, the evaluation gap became clear: benchmark scores looked good, but real conversations exposed weaknesses that synthetic data alone couldn't fix. Shopify's answer was a continuous improvement loop built on production data.
A diagnostic layer for model quality
The first component is an LLM-based judge that scores every conversation across workflow lifecycle facets: whether the assistant understood merchant intent, chose a Flow solution only when appropriate, selected the right components, and gave clear next steps. Instead of a single pass/fail grade, each facet is scored separately. The judge was calibrated against human annotations on hundreds of conversations and validated against production activation rates.
A parallel tagging system classifies workflows along multiple dimensions: triggers, conditions, actions, and third-party integrations. Slice analysis across tags pinpoints where the model struggles, so the team knows exactly what kind of data to add when performance drops.
The patterns revealed were concrete:
- Email workflows caused 25% of failures, prompting email-specific examples
- Diverse condition patterns caused 16%
- Workflow editing, which synthetic data had never covered
Closing the loop weekly
Every production conversation now becomes a potential training signal. Conversations where merchants actually activated the workflow afterward are sampled, scored by the judge, and high-scoring examples automatically feed into the training pool; low-scoring ones are quarantined for human review.
The cycle runs weekly:
- Ingest production conversations
- Score with the LLM judge
- Route high-quality examples into training; quarantine low-quality for review
- Identify gaps through tagged slice analysis
- Retrain and deploy
The approach mirrors reinforcement-style automated research loops in spirit, but applied to production data curation. When benchmarks said the model was ready and production disagreed, this flywheel closed the gap in two weeks.
The next phase: simulation and self-improvement
Keeping pace with new frontier models every few months requires compounding improvements in data, training, and evaluation. Shopify is pursuing three directions.
Simulation environments. A sandbox where the model generates workflows and receives structured feedback on whether they would succeed, without touching real merchants. The model writes test cases and runs them against a simulated Flow environment, enabling verifiable rewards, distillation from stronger teacher models, and on-policy optimization.
From off-policy to on-policy. Current training is off-policy: the model learns from curated examples collected after the fact. Verifiable rewards from simulation enable policy optimization where the model learns from its own generated trajectories, discovering better strategies instead of replicating what it has seen.
From manual calibration to self-improving evaluation. The judge is now manually calibrated against human annotations and activation rates, but merchant shifts, new integrations, and emerging workflow patterns outpace manual recalibration. Automating judge calibration against live production signals is the next evaluation milestone.
Production results and generalization
The fine-tuned flow agent now carries the majority of production traffic. Each stage built on prior ones: the Python DSL made synthetic data generation accurate, production mirroring made the DSL hold up in the real environment, and infrastructure stability made mirroring trustworthy. No single technique was decisive.
The pattern generalizes when four conditions hold:
- The task requires tool calling. The model must reason, act, and incorporate external results rather than emit text.
- The output is a custom DSL absent from pretraining data, whose semantics can be expressed in a language the model already knows.
- A round-trip transpiler is feasible between in-distribution representation and production format.
- A production feedback loop exists. Synthetic data starts the process; real-world data reaches production quality.
Within the broader assistant, the recipe is being applied to other skills: isolate the skill, fine-tune the tool-calling model, and build the improvement loop. Six months of this approach took the system from a rented frontier model to one trained on owned infrastructure with proprietary data, at 68% lower cost — and the deployed version already trails the one retraining behind it.
