Unlocking Real-time Predictions with Shopify's Machine Learning Platform - Shopify

Real-Time Model Serving on Shopify’s ML Platform

Merlin, the machine learning platform Shopify launched in 2022, was built to unify model training across the company. Since that initial rollout, the platform team has been closing the loop on the full ML workflow. Recent additions include Merlin Online Inference for serving models in production, integration with Comet ML for experiment tracking and model registry, Merlin Pipelines for reproducible training workflows, and the Pano feature store, which builds on Feast for both offline and online access to features.

The most demanding of these additions is online inference. Many ML teams at Shopify build models that back user-facing features, where predictions must be returned with minimal delay. Supporting real-time serving across a diverse set of teams, models, and frameworks required more than just spinning up an API endpoint; it demanded a reusable, generalized serving infrastructure.

For teams considering online inference, the trade-offs differ from batch processing in several key ways. Real-time serving incurs ongoing infrastructure cost, stricter latency requirements, and a need for engineers who can run and maintain production services. Before expanding Merlin, the platform team interviewed internal stakeholders to understand their service-level objectives, then designed the system architecture and staffing recommendations around those constraints.

Two Paths to Prediction

Machine learning inference comes in two forms. Batch inference runs on a finite dataset, scheduled periodically. Online inference computes predictions as input arrives, handling a potentially infinite stream of requests. Applications like recommender systems, fraud detection, product categorization, and inbox classification all benefit from real-time predictions, where stale results are significantly less useful.

Because online inference directly touches user-facing applications, latency becomes a primary concern. A slow model can degrade the performance of the broader service that calls it, which can have direct business impact.

Designing Merlin Online Inference

The team set several core requirements for Merlin Online Inference. The serving layer had to work across the machine learning frameworks already in use at Shopify, including TensorFlow, PyTorch, and XGBoost. It needed to deliver minimal processing latency. Production features like rolling deployments, autoscaling, and observability had to be included from the start. Integration with the rest of Merlin, particularly the model registry and Pano feature store, was non-negotiable since users would rely on those existing components. Finally, the path from model to deployed service had to be streamlined so teams could manage their own production deployments.

Rather than design a one-size-fits-all serving system, the team opted to give each ML use case its own dedicated service. These services run on Shopify’s Google Kubernetes Engine clusters, deployed the same way as any other internal service. Each service gets its own Kubernetes namespace and can be configured independently, including autoscaling rules tuned to its own traffic patterns and metrics.

Each inference service loads its model and any required artifacts from the Comet ML model registry. Clients — typically other Shopify core services or Flink streaming pipelines for near-real-time predictions — call the exposed inference endpoint. During inference, the service can pull features from Pano with low latency, as can clients preparing requests before they hit the endpoint.

Every deployment includes a monitoring dashboard with predefined metrics covering latency, requests per second, and CPU usage. Teams can observe the health of their service out of the box, then customize the dashboard for their specific operational needs.

Each Merlin Online Inference service is composed of two main pieces:

• Serving layer: the API that takes a model and returns predictions at an endpoint.
• Deployment: the aspect that determines how the service runs on Shopify’s infrastructure.

The Serving Layer: From API to Production

Merlin Online Inference's serving layer is the component responsible for exposing trained models via a request/response API. Written in Python for accessibility to data scientists and ML engineers, it handles three core tasks: starting a web server, loading the model plus supporting artifacts into memory, and exposing an inference endpoint that maps input features to predictions.

The serving layer lives inside a Merlin Project, a folder in the Merlin mono-repo containing all code, configuration, and tests for a given use case. This structure allows the serving layer to reuse workflow logic from other parts of the ML pipeline. A dedicated Dockerfile (built via Podman) packages the serving layer into an image that is then deployed onto Merlin's Kubernetes clusters.

Two Libraries, Three Approaches

Rather than forcing a one-size-fits-all API, Merlin supports two serving libraries to cover a spectrum of user needs — from zero-code setups to fully bespoke services.

• MLServer: An open-source inference server from Seldon that ships with REST and gRPC interfaces, request batching, and support for the V2 inference protocol. This standardizes communication between different inference servers, improving portability.
• FastAPI: A high-performance Python web framework for teams that need complete control over their API.

To remove the boilerplate burden, the Merlin CLI scaffolds a new serving layer from a cookiecutter template. The user selects their preferred inference library and the machine learning framework for loading their model, and the CLI generates the starter code.

A Custom Model in Practice

With the scaffold in place, implementing a custom servable is straightforward. For example, serving a Hugging Face translation model with MLServer requires creating a class that inherits from mlserver.MLModel and implementing just two methods:

1. Load: Loads the model and any extra artifacts into memory.
2. Predict: Generates predictions from the incoming payload.

Even this minimal code can be avoided. MLServer also provides a pre-built Hugging Face runtime, letting you serve a model using configuration files alone. This low-code/no-code path lets users focus on the model itself rather than API plumbing.

Iteration via Merlin Workspaces

Before deployment, users can validate their serving layer inside a Merlin Workspace—a short-lived, isolated environment defined by its own code, dependencies, and resource requirements. Workspaces support distributed computing per use case and expose a temporary endpoint for debugging and stress testing. Each workspace runs its own instance with selectable infrastructure, Python packages, and serving layer, without the overhead of a full service deployment. Users can also access their API's Swagger page from the workspace to confirm behavior before shipping.

Deploying the Merlin Service

Once validated, the serving layer is bundled with the model, artifacts, libraries, and container image from CI/CD into a production-ready Merlin Service.

Automated Service Creation

The same Merlin CLI handles service creation by leveraging Shopify's internal service infrastructure. This gives Merlin Services access to the same scaling capabilities, tooling, and operational support as any other Shopify production service. A new service is immediately registered in Shopify's Services DB, which tracks all production services and provides teams with views and maintenance tools.

Creation automatically generates a full build-and-deploy pipeline. Merging changes triggers Shopify's Buildkite pipeline, which builds the service image and hands it off to internal Shipit pipelines for deployment to Shopify's Kubernetes clusters.

Per-Environment Configuration

Each Merlin Service is created with two configuration files—one for production, one for staging. These define CPU, memory, GPU, autoscaling rules, artifact paths, and project metadata. Separate settings per environment let teams optimize resources and cut infrastructure costs. The staging setup is especially useful for validating new model versions or settings before promotion to production. A typical configuration might specify an MLServer-based classification model with a minimum of 3 replicas (scaling to 10), each allocated 2 CPUs, 32 GB of memory, and an nvidia-tesla-t4 GPU, pulling its model from the registry at startup.

Roadmap: Ensembles, Monitoring, and Retraining

The complete Merlin flow now looks like this: a project's image is built, used in training, and the resulting model is saved to the registry. When online inference is needed, a serving layer and service are created and deployed, allowing users to iterate and push new model versions as they go. The platform is currently evaluating several open-source tools, including Ray Serve, Seldon Core, and BentoML, to support orchestrated multi-model inference. It is also planning platform-based drift monitoring to replace manual workflows, and automated continuous training pipelines to handle models that need frequent retraining as prediction data grows. Even in its current iteration, the online inference infrastructure is already giving Shopify's data teams the low-latency, scalable path to real-time predictions they need.