Using Rich Image and Text Data to Categorize Products at Scale - Shopify

Rethinking How We Identify What Merchants Sell

Shopify has grown considerably since we first discussed product categorization on this blog. We now support millions of merchants selling billions of products across a wide range of industries. With that growth came the need to re-examine our existing categorization model and verify that it still understood what our merchants were selling well enough to power the products we build for them.

Our evaluation centered on two primary metrics. The first was prediction quality: precision, recall, and accuracy measured against a hold-out set. The second was coverage, which we define as the ratio of products that receive a prediction to the total number of products. Our legacy model filtered out predictions below confidence thresholds, so coverage tells us how often we're actually helpful versus silently declining to answer.

We also considered how predictions would be consumed, since some use cases demand low-latency, real-time inference. After weighing these factors, we saw clear room for improvement: the old model relied exclusively on English text features, which created significant blind spots. The new model we built increases leaf precision by 8% while doubling coverage.

Why Precise Categorization Matters

The business case hasn't changed: merchants sell diverse products across multiple channels, and understanding what they sell is foundational to building better shopping experiences. Categorizing products into a standard taxonomy enables features like cross-channel search and discovery, plus insights that support merchant marketing efforts.

We continue to use the Google Product Taxonomy (GPT) as our organizing structure. GPT contains over 5,500 categories arranged as a hierarchical tree rather than a flat label set. Both the sheer number of classes and the parent-child relationships between them make this a substantially harder modeling problem than standard flat classification.

Available Features and How We Vectorize Them

Merchants provide rich product data through the Shopify admin: title, description, vendor, product type, collection, tags, and product images. None of this arrives in a structured format that maps neatly to taxonomy nodes. Two merchants selling identical items may use entirely different values for "Product Type," which gives merchants flexibility but complicates cross-store indexing.

Broadly, our features split into two types: text and image. Since raw text and images aren't directly usable by most machine learning models, we evaluated vectorization strategies and settled on transfer learning. We tested several pre-trained models for both modalities, balancing model performance against computational cost. Our final choices:

• Multi-Lingual BERT for text embeddings
• MobileNet-V2 for image embeddings

This approach also gave us flexibility to incorporate several design principles described in the architecture section below.

Architecture: A Multi-Task, Multi-Class Approach

Our previous attempts taught us two lessons about hierarchical classification. First, preserving the multi-class nature of each taxonomy level is extremely beneficial—Level 1 has just 21 labels while Level 3 exceeds 500. Second, learning parent nodes helps predict child nodes: it's far easier to predict "Dog Beds" once you've determined the product belongs to "Dog Supplies."

We framed the problem as a multi-task, multi-class classification problem to embed those lessons directly into the model. Each level of the taxonomy is treated as a separate classification task, and the output from each level gets fed into the prediction of the next level.

During the forward pass, we embed raw text and image features using our pre-trained models, then pass those embeddings through hidden layers to produce a multi-class output for Level 1. That Level 1 output, along with the original embeddings, is concatenated and passed to subsequent hidden layers to predict Level 2. This feedback loop continues through all seven taxonomy levels.

Five design points are worth calling out:

1. Seven output layers, each corresponding to a taxonomy level and each with its own loss function.
2. Parent nodes influence child node outputs during the forward pass.
3. Backpropagation combines all seven losses in a weighted fashion to produce gradients, so lower-level performance can nudge weights in higher layers.
4. No hard hierarchy constraints during training. The model is allowed to predict Level 2 as "Pet Supplies" even if it predicted Level 1 as "Arts & Entertainment." This freedom lets accurate child predictions correct wrong parent predictions.
5. Class weights handle imbalance. The dataset is highly skewed, and penalizing errors on under-represented classes helps us train a classifier that generalizes better.

Training at Scale

Shopify's scale is an advantage here: we have hundreds of millions of merchant product observations to learn from. But this model has over 250 million parameters, and training on a single machine with even GPU acceleration would take multiple weeks. We needed to reduce training time without sacrificing quality.

We chose a data parallelization strategy, chunking the training dataset and assigning one machine per chunk. The model was built and trained using distributed TensorFlow with multiple workers and GPUs on Google Cloud Platform, with several optimizations applied to keep resource utilization high.

Inference with Hierarchy Enforcement

Our training setup deliberately allows the model to ignore the taxonomy's parent-child constraints. While that freedom improves training dynamics, consumers need predictable behavior at inference time. We added post-processing logic to enforce hierarchical consistency.

1. Run the raw forward pass to obtain seven confidence arrays, one per taxonomy level.
2. Take the highest-confidence category at Level 1 as the Level 1 prediction.
3. Restrict Level 2 to the immediate descendants of that Level 1 prediction, and pick the child with the highest confidence.
4. Continue recursively through all levels down to Level 7.

We implemented this recursion as TensorFlow operations and wrapped it—along with the trained Keras functional model—into a single Keras subclass model. That gives us one TensorFlow model object capable of serving both batch and online inference, with hierarchy enforcement built in.

The resulting product is already in use by multiple internal teams and our partner ecosystem to build derivative data products.

Evaluating the Model

The team tracked a suite of hierarchical metrics — accuracy, precision, recall, F1, and coverage — to measure the new model's performance. Alongside gains in every metric, the model also classifies products in multiple languages, removing the English-only limitation of its predecessor. That multilingual support matters for a platform aiming to serve merchants worldwide.

Not every product receives a prediction at every level of the taxonomy. The system applies confidence thresholds at each level, filtering out low-confidence predictions so that only the most reliable results are exposed to merchants. The image below shows how this works in practice:

In the dog bed example, the first three levels of prediction all clear the confidence bar and are surfaced. The fourth level does not meet the minimum threshold, so it stays hidden. This approach means a product might have predictions at only one, two, or three levels depending on how confident the model is at each step.

Balancing these metrics required careful tuning. Raising hierarchical precision, for instance, could be done at the cost of lower coverage. The team had to weigh those trade-offs against business priorities, focusing on reducing negative merchant experience and friction rather than chasing purely numeric gains. Metrics were a useful signal, but the team also ran spot checks and manual QA on predictions to catch problems that aggregate numbers could mask.

One notable case was model performance on sensitive categories such as “Religious and Ceremonial.” Overall metrics can look strong while hiding poor behavior in small corners of the taxonomy — exactly the areas where a wrong prediction creates the most merchant friction. The team manually tuned confidence thresholds in these areas to ensure high performance. The recommendation for anyone shipping a machine-learning-powered consumer product is to adopt a similar practice: let business context guide model tuning, not just headline numbers.

Results and Next Steps

The upgrade delivered an eight percent increase in precision while nearly doubling coverage. The new model produces more accurate predictions for a much larger set of products. But the team sees room for further improvement in two main areas.

Data quality. The labeled product dataset is large and rich, but highly imbalanced. Standard techniques like class weights and over/undersampling can help, but the team believes fresh data collection is also needed in under-represented areas. As Shopify's merchant base grows, the taxonomy keeps expanding into new product categories, so the data pipeline needs to keep pace.

Merchant-level features. The current model uses only product-level signals. There is additional signal available at the merchant level: a store called “Acme Shoe Warehouse” strongly hints at the types of products it sells. Incorporating these broader features could improve predictions further.