Choosing the First Machine Learning Problem
Shopify’s data science team shipped its first machine learning product five years ago. Since then, the company has integrated ML across many of its systems. Getting to that point meant learning how to identify the right problems to solve first, how to get models into production, and eventually how to scale a portfolio of models across the organization.
According to the team, the problems you choose to solve first have a disproportionate impact on the growth of your machine learning portfolio. The earlier decisions set a template for everything that follows.
Find a Problem Users Actually Care About
The ideal first problem is one your users care about on a daily basis. That ensures consistent usage and fast feedback. The key to identifying such a problem is having deep domain context: understanding both business goals and user needs. At Shopify, data scientists are embedded within product and commercial lines so they can act as partners in decision making.
This approach led Shopify to its first problem: order fraud detection. Since merchants care about sales above all, and the existing rule-based solution was a known pain point, fraud detection was a natural fit. 
Verify Your Data Is Trustworthy and Accessible
Having data isn’t enough; it needs to be both reliable and accessible. If data scientists have to request moves from an S3 bucket through an operations team, that barrier can stall progress. Shopify had 10 years of transaction data available through its internal data platform, with easy access via tools like Trino. The technology is less important than having trustworthy data that can help you understand the problem at hand.
Map Your Model’s Downstream Dependencies
No model exists in isolation. Understanding the user workflow around your problem should shape the conditions of your target. In fraud detection, fulfillment is a downstream dependency. Merchants won’t fulfill high-risk orders, so fraud had to be identified before fulfillment. A model that detects fraud after an order ships is useless no matter how well it performs.
Understand the Existing Solution
Before adding machine learning, you must understand what already exists. Dig into the code, data, and domain experts to find the current solution’s weaknesses. The existing rule-based fraud system had a high false positive rate, flagging any order where billing and shipping addresses differed. Merchants had to investigate each flagged order, consuming time they could spend growing their business. Worse, some merchants cancelled good orders due to the noise.
Optimize for Product Outcomes, Not Metrics
The temptation is to optimize for accuracy, precision, or recall, but those scores don’t matter if the user experience doesn’t improve. 
Shopify’s guiding principle was merchant success: helping merchants make valid sales. A model optimized for zero fraud would flag every order, which technically achieves the fraud goal while preventing all sales. Instead, Shopify’s model optimized for the highest number of valid merchant sales, setting thresholds that reflected real user needs.
Moving From Model to Production
With a model built, the next challenge is bringing it to production in a way that can scale. Success depends on the strength of the underlying data engineering practices.
Build Complete Pipelines
You need well-defined pipelines for all stages of predictive modeling. In training, mixed-up data cleaning (was null replacement done consistently?) erodes trust. Shopify labeled input data, transformations, and features clearly to keep its pipeline clean. The same goes for verification and testing: capturing rich metadata about which model and dataset was used enables reproduction of metrics and traceability of bugs. Good data engineering reduces repetitive work and helps establish model trust with stakeholders.
Choose Deployment Based on Context
Deployment choices depend heavily on the problem and product context. Two factors shaped Shopify’s decision:
- Expected volume. Shopify had to handle everything from normal traffic to peak moments like Black Friday. This led to deploying the fraud model on Shopify’s core Ruby on Rails stack, since high-volume services already have resources dedicated to keeping them operational.
- The user-product commitment. Checkout speed is critical for merchants, so fraud detection had to happen in real time without slowing the checkout flow. A great model that drags down checkout would create a new problem while solving another.
Shopify initially shipped a simple logistic regression model. That simplicity let them move quickly and demonstrate impact. The model beat its baseline and grew to process millions of orders per day. That first win became an internal proof point for adopting machine learning more broadly.
Scaling From One Model to Many
Once you have a model in production and a team that trusts the process, the question becomes: how do you multiple the portfolio without multiplying toil?
Build Trust That Holds Up
To scale, some initial engineering practices need to become fully baked into the model lifecycle. Shopify implemented three processes to keep its models trustworthy over time:
- Input and output reconciliation. Verify that training data definitions and measurements match inference-time data. Shopify found many bugs through this simple check, confirming that for the same input, the model predicts the same thing every time.
- Production backtesting. Run the model in shadow mode for a cohort of real users, as if it’s powering an actual experience. Observing predictions against real-world data — without exposing users to changes — helps reveal how the model behaves live and gives a mechanism for comparing candidates.
- Monitoring. Log daily, run systematized change logs to avoid stale model assumptions. For example, Shopify once found that mobile orders correlated strongly with fraud. Over time, mobile became the dominant shopping method and the correlation broke. Monitoring catches these distribution shifts, either through automated systems or dedicated humans.
Code Standards into the Platform
With a small team of 10 data scientists, standards can live in Slack or a shared document. As teams and portfolios grow, those aren’t enough. Shopify began encoding best practices directly into its internal platform. For instance, since backtesting is required before production, the platform simply fails deployment for any model that hasn’t been backtested. This turns guideline-dodging into a technical impossibility without manual override.
There's a risk of abstracting too early, and Shopify notes it spent a few years refining practices from its first fraud model and a few others before committing them to platform code. Only after trial and error did best practices become encoded.
Automation as the Scaling Lever
A solid foundation only gets you part of the way. If model monitoring, versioning, and deployment had to be done by hand on every iteration, the team would still be refining its first model. The useful question to ask is not "how do I build more models?" but "how do I scale beyond the hours I invest?" That means thinking in terms of model operations: scheduled runs, automatic checks, and versioning, eventually leading toward automated deployment.
Shopify built all of this into its infrastructure so that jobs execute on a recurring schedule for every merchant—daily, weekly, or on whatever cadence is appropriate. Humans remain in the loop, but their role shifts to investigating flagged anomalies rather than shepherding each run. Automating the operational side of machine learning is what gives data scientists the room to focus on building new models instead of keeping existing ones alive.
The result of this approach is faster deploy and retrain cycles. What began as a single order-fraud detection model became a repeatable pattern applied to Shopify Capital, product categorization, the Shopify Help Center search, and hundreds of other products. The path from one model to a hundred follows the same playbook, applied repeatedly.
A Full-Stack Discipline
The playbook describes the process, but the people matter just as much. Scaling machine learning output is ultimately a cross-functional problem: it requires data scientists and data engineers to work from the same roadmap toward the same goal. No matter how the organization is structured, both roles are instrumental to the success of a machine learning portfolio. That alignment is what turns a set of model-building steps into a durable capability for delivering business value.
