5 Steps for Building Machine Learning Models for Business - Shopify

Machine Learning With a Business Mindset

Most of what’s written about machine learning focuses on the technical side: model architectures, training tricks, and optimization. Far less guidance exists on the equally important business context — when ML makes sense, how to start, and how to keep a model useful without disrupting the product it powers. At Shopify, we’ve distilled this into five practical steps that help teams get models into production and actually drive impact.

Step One: Check Whether ML Makes Sense Right Now

Before committing engineering resources, decide if a model is the right investment at this stage. It’s tempting to build something sophisticated, especially for a product expected to feel “smart.” Two questions help ground that decision.

Is This a New Product?

Early product development is resource-constrained: finding product-market fit, driving engagement, and collecting feedback should come first. Delaying ML frees up time for those fundamentals. But do plan the data strategy now. The future data flywheel — what data you’ll collect and how it will feed a model later — is worth designing from day one. If the product already solves a real problem, ML can then extend that value.

How Well Do Simple Methods Work?

Start with a heuristic before anything learned. Simple rules are faster to build, easier to debug, and frequently perform far better than expected. They also give you a baseline for measuring a more complex system later. A rule-based approach is viable in roughly 90 percent of cases for initial problem solving.

Step Two: Keep It Simple

For a first model, resisting complexity matters. Extra features and trendy architectures can add accuracy, but they also lengthen build time, complicate the system, and slow down onboarding. A simpler model gets the full pipeline running end-to-end sooner, and systems complexity is easier to add than to remove. Starting plain doesn't preclude sophistication later.

• Start with interpretable models. Shallow decision trees, linear regression, and logistic regression train quickly and are easy to explain — a major advantage for stakeholder buy-in. In practice, we built a lead-scoring system where random forest matched a more complex ensemble, so we shipped the simpler model for speed and interpretability.
• Start with basic features. Use a small, known-relevant feature set to get fast initial feedback. Domain insights help here. For our shop industry classifier, we knew a store’s product weights correlated with its category — furniture is heavier than apparel — so we used that and similar signals before engineering more.
• Use off-the-shelf solutions when apt. Deep learning is needed for strong performance on images, video, audio, or text. For those, pre-trained models get you there quickly. A pre-trained word embedding feeding a logistic regression can be enough for a release; fine-tuning comes later if performance demands it.

Step Three: Define Measurements Before Optimizing

Tuning hyperparameters in the first iteration is a common trap. With many parameters to adjust, optimizing too early wastes compute and masks underlying problems. Answer these questions first.

How Will Incremental Business Value Be Measured?

Good benchmarking needs two parts — both the metric and an explicit plan for how changes translate to impact.

Choose a metric aligned with business objectives. You must understand what the number really means. In fraud detection, 98 percent accuracy is not a win — it likely reflects a rare-class problem where the model output is meaningless for catching actual fraud. The right metrics vary by context, not by what’s easiest to report. Baselines from your earlier heuristic work anchor the comparisons.

Connect that metric to the business outcome. Design a way to quantify effects. If accuracy rises from 0.8 to 0.85, does the user actually notice? Are those extra points worth the added complexity? This analysis should drive whether to continue tuning, change the metric, or stop investing.

Can You Explain the Model's Tradeoffs?

Impressive performance demands scrutiny. In our experience, about 95 percent of “magical” results trace back to a systemic issue: the wrong metric, data leakage, or a subtle data imbalance. Inspecting model output, not just its score, is how such risks surface.

When the metric is stable and its business impact understood, optimization has proper guardrails: you stop improving when there is no measurable gain. And by regularly looking at what the model misses or over-predicts, you catch issues early rather than polishing a fundamentally broken system.

Step Four: Plan for Model Iteration

Models are living systems. They get retrained periodically, their architecture changes for better performance, and their features shift as business needs evolve. Rollouts must never cause regression against the baseline. A few practices keep that process safe.

Invest in Side-by-Side Evaluation

Building a pipeline that can evaluate two or more model variants concurrently prevents bad promotions. Two solid evaluation methods:

• Offline staging: When you can score data without hitting production (e.g., a classifier with available labels), run the challenger silently for a period first. For our shop industry classifier, a major update sat in staging for weeks before promotion. Offline testing avoids user-facing regression.
• Online A/B testing: Show a random subset of users the new model and compare user-facing metrics against the baseline. For recommendation systems where the metric is engagement, this gives a clear read on lift or loss.

Verify That Comparisons Are Fair

Before trusting a side-by-side comparison and making a roll out decision, ensure the evaluation conditions match. If the class balance differs between the evaluation set and production, the results won’t reflect reality. Changing the evaluation population can also introduce bias. When fair comparison isn't possible, reweight or adjust the metrics — or change the evaluation setup — before deciding which model to promote.

Performance Metrics Need Scrutiny Before You Trust Them

A single evaluation run rarely tells the full story. Model performance shifts with random parameter initializations, the particular train/test split, or minor hyperparameter tweaks. Check whether results stay consistent across runs and over time, and confirm the model behaves well across key population segments. Inconsistent performance is often an early signal of deeper problems. If some segments matter more to your business, consider reweighting the metric so those segments have a stronger influence on the score.

Your Comparison Process Can Mislead You

Optimizing against a fixed test set risks overfitting to those exact examples. The solution is to build safeguards into your comparison framework. Cross-validation, rotating test data, keeping a separate holdout set, regularization, and repeating tests when random initialization is involved all reduce the chance that apparent gains are just artifacts of your evaluation setup.

Model Drift Cuts Both Ways

Stability over time is an under-appreciated concern. Suppose your model is retrained quarterly and its metric steadily improves. Overall performance is up, but that does not mean every user saw the same positive change. Some individuals may see their predictions flip even as the aggregate improves. Those users can end up with a worse experience than anyone on the team expected.

Imagine a model that decides whether a user is eligible for funding, and the user can see that eligibility status. If the status fluctuates between retrains, users get frustrated and lose faith in the product. In such cases, stability may be worth more than marginal performance gains — you might even fold stability directly into your performance metric.

Measuring this effect is the first step toward managing it. The causes vary by context and are not always a form of overfitting. Use this checklist to reduce the risk:

• Understand the costs of model changes. Weigh improved performance against the impact of changed predictions and the engineering work needed to handle them. Avoid major overhauls unless the performance upside clearly covers the cost.
• Prefer shallow models over deep ones. In classification, for instance, a training set change is more likely to shift a deep model’s decision boundary in local regions. Reserve deep models for cases where the performance gain justifies their instability.
• Calibrate the output, especially for classification and ranking systems. Calibration makes distribution shifts visible and reduces their impact.
• Watch objective function conditioning and regularization. A poorly conditioned model can swing its decision boundary wildly from small training changes.

What Decides Whether an ML Project Succeeds

These factors interact, and any one can sink a business-focused machine learning initiative. Five points deserve particular attention:

1. Question whether machine learning is needed now. For a new product, launch a baseline solution first and layer ML in later when you understand your data and users better.
2. Keep it simple. Simple models and feature sets iterate faster and are easier to interpret — both matter especially for a first ML product release.
3. Measure before you optimize. Understand what your metric really tracks and how it connects to business goals. Make sure you know the tradeoffs your model is making.
4. Plan for model iteration. Improvements will come incrementally, so establish a comparison process that fairly judges each new version against the incumbent.
5. Account for stability over time. Think about how prediction changes affect users and factor that into your iteration choices.