General-purpose ranking models, like those used for product discovery on marketplaces, are trained to predict whether a user will click an item based on past behavior. For a consumer shopping for, say, a "birthday gift for dad," that approach often misses the mark. A query like this carries seasonal urgency and a personal relationship context that a standard engagement-based model can't easily encode. The result is a list of popular, but generic, products rather than a thoughtful shortlist.

Shopify's product discovery team faced this problem directly when building search for its consumer app. Their models had to move beyond simple relevance and toward understanding the intent behind a query—especially when that intent changes faster than the model can be retrained.

Modeling intent as a real-time signal

The core technical challenge is latency. A user's purpose can shift within a single session—from browsing to buying, or from broad inspiration to a specific need. Batch-updated user embeddings or static query categorizers fail to capture this. Shopify's solution was to treat intent as a real-time feature that is continuously updated in the model's serving path.

They built a system where the search query and recent user actions are fed into a lightweight transformer encoder at inference time. This encoder produces a short intent vector that is then combined with a slower-updating profile vector (based on long-term purchase history) and a product vector. The fusion of these three vectors happens online, per request, allowing the model to adapt its ranking output immediately after a user adds an item to cart or narrows their filter range.

The training objective also reflects the difference between consumer and merchant search. Instead of optimizing only for click-through rate, they used a multi-task loss that combines engagement prediction with a post-purchase satisfaction head. This head predicts, at the moment of the search, the probability that a shown item will be returned or rated poorly, penalizing rankings that surface high-click, low-quality products for intent-driven queries.

Key engineering trade-offs

Putting a transformer in the serving path introduces cost. To keep p95 latency acceptable on production traffic, Shopify made specific architectural choices:

  • Caching: Product vectors are precomputed and stored in a feature store; only the query and session context are passed through the online encoder. This decouples the heavy embedding computation from the per-request path.
  • Pruning: The candidate set retrieved from the search index is limited to roughly 1,000 items using a fast lexical and vector retrieval hybrid, so the expensive scoring model runs only over a shortlist rather than the full catalog.
  • Distillation: A large offline teacher model, trained on several weeks of logs with full context windows, teaches the small online student encoder. This allows the student to retain nuanced intent recognition without the computational overhead of the teacher.

The team also introduced an adaptive exploration mechanism. For a small fraction of traffic (tunable between 1-5%), the model deliberately blends in results that are not the top-ranked by the fusion score. This randomized disturbance, paired with a contextual bandit, lets them gather labels for under-served intent categories like "gift for" or "outfit for event" without permanently degrading the search experience for the majority of users.

Measuring success beyond CTR

Because their loss function includes post-purchase satisfaction, the primary offline metric is not click-through rate but return rate weighted by search position. In their experiments, the new model reduced the probability of a returned item appearing within the first three results by 18% compared to a pure engagement-based baseline.

Online A/B tests further tracked an unusual proxy: session-to-purchase ratio with the time to first click shortened. While CTR on top results were comparable to the old model, the intent-aware model produced longer click chains with more consistent progression to checkout, indicating users found what they were actually seeking rather than stopping at the first attractive thumbnail.

Latency impact was acceptable in production, with p95 search response time remaining under 250ms, a tight budget that forced the team to move all re-ranking logic into a single compute graph with parallel feature fetch.

The role of query understanding

A significant part of the engineering effort went into parsing the query itself. Rather than relying on a single text encoder, the model uses a dual-encoder structure for the query: one branch processes the raw token sequence, while the other processes a structured intent parse. That parse tags segments such as [RECIPIENT: dad], [OCCASION: birthday], and [BUDGET: under $50].

These tags are produced by a lightweight sequence tagger that runs upstream of the main encoder and is trained on annotated session data. The tagger runs at a fraction of the cost of the main model and only triggers the budget and occasion tokens when they materially changes the routing of the candidate set. For plain navigational queries like "running shoes," the tagger outputs little, and the system falls back to a standard product relevance path.

This dual-branch approach proved critical for generalization. The model could correctly handle zero-shot intents—like "what do i get someone who just moved?"—because the occasion tag activated appropriate filter logics even though such phrasing never appeared in the training set.

The combined system, now serving all consumer app search traffic, demonstrates that real-time intent modeling is feasible for e-commerce at scale when latency budgets are strict. The key lesson from Shopify's implementation is not the particular encoder architecture, but the decision to treat intent as a volatile feature updated per-query rather than a static attribute revised weekly.

Turning search into intent with streaming embeddings

Shopify's Storefront Search has moved beyond simple keyword matching into semantic search, where the platform tries to understand what a shopper actually means rather than just what they type. That shift depends on embeddings — numerical vector representations of text and images in a high-dimensional space that let the system measure similarity between a query and product content.

Why the pipelines run in near real time

Those embeddings don't arrive by batch job. Shopify processes roughly 2,500 embeddings per second — around 216 million per day — across image and text pipelines, with updates flowing as merchants create or modify content. The infrastructure sits mostly on BigQuery, with Google Cloud's Dataflow powering the streaming layer for its native streaming support and integration with the wider Google Cloud ecosystem.

The image embedding pipeline follows a straightforward sequence: the model loads at startup, the pipeline listens for events signaling a newly created or updated image, then the image is downloaded, loaded into memory, and resized before inference runs. After postprocessing, the resulting embedding fans out to both a data warehouse for offline analytics and an output topic for downstream consumers like Storefront Search.

A batch approach would be simpler, but merchant expectations rule it out. When a seller edits a product or uploads a new image, they want that reflected on their storefront immediately. Shopify's data indicates that fresher embeddings from a streaming design improve the odds of a sale, which justifies the added pipeline complexity.

Tuning memory on Dataflow workers

The initial deployment ran on n1-standard-16 machines with T4 GPUs, but image processing pushed workers into out-of-memory errors. The first fix — moving to n1-highmem-16 machines — raised available memory from 60GB to 104GB and worked for a while, but it also added 14% to cost. The real solution required understanding how Dataflow's Python runner uses worker hardware.

Dataflow spawns one process per core, so on a 16-core machine that means 16 processes, each with 12 threads by default. With roughly 192 threads all processing elements concurrently, the worker could hold close to 192 images in memory at once. The number_of_worker_harness_threads option provides a lever to change that.

Dropping the thread count to 4 naturally cut preprocessing throughput, but the pipeline was already GPU-bound — images were accumulating faster than the GPU could score them. Inference throughput barely moved. Setting that parameter reduced the memory footprint from hovering near the 104GB ceiling down to roughly 40GB, a ~2.6x decrease. That let Shopify move back to n1-standard-16 machines and eliminate the extra 14% cost.

Model loading trade-offs

Apache Beam's inference abstraction has two main pieces: ModelHandler, which defines the ML model, and RunInference, the transform that produces embeddings. By default, each of the 16 worker processes loads its own model instance onto the GPU, which boosts parallelism but consumes GPU memory 16 times over.

Beam's ModelHandler supports sharing a model across processes, which loads it once and lets other processes query it. Memory consumption drops sharply, but so does throughput, and Shopify abandoned that path. Controlling process count directly isn't possible, and forcing a single SDK container per worker via experiments=no_use_multiple_sdk_containers degrades throughput too much. In the end, the embedding models were compact enough that Shopify kept Dataflow's default configuration.

Batching against a GPU bottleneck

CPU utilization past 80% invites thrashing, but GPU saturation is the goal — context switching on a GPU streaming multiprocessor is essentially free. The real constraint is transferring data between host and device, which is nearly always the bottleneck in this kind of streaming pipeline. Batching is the answer, and Beam's batch_elements_kwargs method lets users define batch sizes for GPU submission.

Except batches of 1 were still reaching the GPU. RunInference internally uses the BatchElements transform, which can batch within the current bundle or across bundles using a stateful implementation. Bundles — groups of elements processed together — are sized by the Dataflow Runner, not the user. With bursty input topics, elements were landing in bundles of 1, making in-bundle batching ineffective. Stateful batching with max_batch_duration_secs can guarantee batch size but forces a shuffle and adds latency.

The pipeline still managed to saturate the GPU because each process loads its own model instance, generating enough parallelism to offset the frequent host-device transfers. Shopify chose in-bundle batching over the latency penalty of stateful batching, and continues to investigate best practices in this area.

What's next for ML assets

The embedding pipelines serve as reusable ML building blocks, and Semantic Search is the first major consumer. Shopify is now working with internal teams to identify other problems these shared primitives could solve. Related work is documented in the company's post on building the Shopify Inbox message classification model, and Google Cloud's Dataflow ML documentation covers getting started with similar pipelines.