Product Understanding at Shopify: A Shift from Category Labels to Full Product Intelligence
Shopify’s catalog spans everything from handmade goods to heavy machinery. To power search, discovery, and merchant tools across that range, the company has moved well beyond simple category tagging. Its current system, built on the Shopify Product Taxonomy and Vision Language Models (VLMs), now handles more than 30 million predictions per day, covering category assignment, attribute extraction, and content safety checks.
From TF-IDF to Two-Stage Prediction
Shopify’s first classification model, deployed in 2018, was a logistic regression over TF-IDF features. That approach worked for straightforward cases but broke down as product diversity grew. By 2020, the team added a multi-modal model that combined image and text inputs, improving accuracy where either modality alone was too ambiguous.
Despite these gains, category prediction alone was insufficient by early 2023. The platform needed more granular understanding, consistent taxonomy across surfaces, category-specific attribute extraction, richer metadata (like simplified descriptions and content tags), and stronger content safety. The arrival of capable open-weights VLMs allowed Shopify to consolidate these requirements into a single stack.
The Current Foundations
The Shopify Product Taxonomy is the organizing layer. It maps over 10,000 product categories and more than 1,000 attributes across more than 26 verticals. Key design points include:
- Hierarchical categories, e.g.,
Furniture > Chairs > Kitchen & Dining Room Chairs. - Category-specific attributes, so only relevant fields are generated per product type.
- Standardized values for consistency with room for customization.
- Cross-channel compatibility via provided taxonomy cross-walks for multi-platform selling.
The Vision Language Models sit on top of this structure. Unlike the earlier image-plus-text pipeline, a VLM processes both modalities jointly. That provides genuine multi-modal reasoning, zero-shot classification of unseen product types, natural language output for metadata generation, and contextual awareness of how an item is used or styled.
In production, the system runs a two-stage prediction loop. The first call predicts the category and generates a simplified description; the second call uses that category context to extract the relevant attributes.
Keeping Inference Fast and Cost-Effective
Shopify has moved through several VLM generations since the architecture launched. The progression from LLaVA 1.5 7B to LLaMA 3.2 11B and now Qwen2VL 7B has delivered steady prediction quality gains without increasing operational cost. Each candidate model is evaluated against the existing pipeline on both accuracy and compute efficiency before adoption.
FP8 Quantization
The current Qwen2VL deployment uses FP8 quantization. That reduces the GPU memory footprint and enables more efficient in-flight batching, all with minimal accuracy loss.
Dynamic In-Flight Batching
Instead of waiting for a fixed batch size, the system groups requests as they arrive. Batch composition adapts in real time, product updates are processed immediately, and new batches form while earlier ones are still executing. This keeps GPU idle time low while balancing latency and throughput. The approach is built on NVIDIA Dynamo.
KV Cache Optimization
A key-value cache stores previously computed attention patterns, streamlining token generation during the two-stage process where both category and attribute outputs are produced sequentially.
Transactional Consistency
The Dataflow orchestration layer enforces consistency across the two dependent model calls. Both category and attribute predictions must succeed; partial failures trigger automatic retries, and output is validated against taxonomy rules before storage and notification. Monitoring covers prediction quality and system health continuously.
Training Data Built on Multi-LLM Arbitration
Annotation quality is the ceiling on system reliability. Shopify’s pipeline uses a multi-LLM annotation scheme where several large language models independently evaluate each product. Structured prompts keep output aligned with taxonomy standards. When the models disagree, a dedicated arbitration system—using specialized judge models—resolves the conflict with explicit ruling logic for edge cases.
A human validation layer handles complex novel product types and feeds corrections back into the loop. Regular quality audits keep the annotation standards and training distribution representative of the live catalog.
Measured Impact and What’s Next
Merchants see an 85% acceptance rate on predicted categories, which translates to better catalog organization, improved search relevance, more precise tax calculations, and reduced manual tagging effort. On the buying side, the structured attributes support clearer product information, sharper search results, and better recommendations. Platform-wide, the system reports a doubling of hierarchical precision and recall over the earlier neural network baseline. The structured attribute set now covers every category, which also strengthens automated content screening and platform trust.
Shopify plans to keep extending the taxonomy and model stack. New VLM architectures will be evaluated as they appear. Catalog coverage will broaden into more specialized categories and multi-lingual descriptions, and inference pipelines will be tuned for higher throughput. The largest structural change is a migration from the tree-based taxonomy to a Directed Acyclic Graph (DAG). That will let a single product legitimately belong to multiple category paths and more cleanly represent cross-category items. Finer-grained metadata work is also planned: measurements, specifications, materials, and design elements will get deeper extraction coverage across more taxonomy branches.
From classifying to understanding
Shopify's product classification system has evolved from a simple category-based pipeline to a comprehensive understanding layer that powers search, tax, analytics, and other merchant-facing features. The shift is not just about labeling products correctly; it is about extracting rich, structured metadata that can be reused across the platform. The key technical driver has been the integration of Vision Language Models with Shopify's existing standard taxonomy.
Whereas earlier iterations depended on two-stage text-based classification, the new system treats images and text as complementary signals. A fine-tuned Vision Language Model processes product images and generates candidate metadata, which is then validated against the official taxonomy. This grounding step prevents the model from inventing categories or attributes that do not exist in the structured tree, keeping outputs consistent with downstream consumers that rely on strict schema adherence.
The architecture: VLM plus grounding
The core pipeline plays to the strengths of both components. The Vision Language Model provides broad, flexible understanding of a product from its visual appearance, including attributes that are hard to capture from title text alone, such as material, shape, or color. The grounding layer then maps those free-form candidates to canonical taxonomy nodes and attribute values.
This design avoids the brittleness of a pure text classifier that struggles with sparse or uninformative titles, and equally avoids the chaos of an ungrounded generative model that produces ad hoc labels. By constraining the final output to a known ontology, Shopify retains the interoperability of structured data while gaining the perceptual breadth of modern vision models.
Scale and production considerations
Operating at Shopify's scale required significant model efficiency work. The system runs with 30 million daily predictions and maintains coverage over billions of historical products. To reach that throughput, the engineering team made deliberate trade-offs in model size, quantization, and batch serving, ensuring that the Vision Language Model could operate within production latency budgets without sacrificing the grounding step.
That scale is meant to be invisible to the merchant. The predictions feed directly into search indexing, tax code assignment, and analytics pipelines, which means any degradation in accuracy has immediate downstream effects. The team emphasizes that the real-world value of the system is not the model itself but the reliability of the entire loop: seeing a product, extracting every attribute used elsewhere on the platform, and persisting that understanding in a queryable form.
The broader lesson Shopify draws from this deployment is that Vision Language Models are no longer confined to demo-grade image captioning. When paired with a strict grounding mechanism and a well-known ontology, they can replace heuristic pipelines for complex, multi-attribute classification at large scale, handling data volumes that previously required hand-tuned rules or brittle surface-level features.
