Multimodal LLMs at the Core of Shopify’s Global Catalogue

Shopify’s commerce ecosystem spans millions of merchants who describe billions of products in their own unique way. That diversity drives commerce but creates a serious technical hurdle: machine understanding requires standardized, structured data, and unstructured merchant input is fragmented at best. At the ICLR 2025 Expo in Singapore, Shopify engineers shared how they’re addressing this with Global Catalogue, an intelligence layer that unifies and enriches product data using multimodal LLMs.

Presented by Audrey-Anne Guindon, Jonathan Ohayon, Ali Khanafer, and Yang Liu, the talk walked through the technical underpinnings of this system—from data ingestion to model training and the infrastructure that powers tens of millions of daily inferences.

Why Product Data is Hard to Standardize

The root problem is that Shopify was designed around merchant autonomy. Each shop can define its own product schema, and merchants describe items with whatever level of detail they choose. While that flexibility lowers barriers for entrepreneurs, it results in data that machines struggle to parse:

  • Unstructured data: Most information lives in free-form text rather than standardized fields. One merchant may provide a full novel of description; another only a title and price.
  • Schema heterogeneity: Attribute names and values are merchant-defined, producing non-uniform data structures across the platform.
  • Data quality issues: Typos, missing values, misclassifications, and stray content are common. Fields often hold the wrong value—like brand mentioned only in the title, not the brand attribute.
  • Multimodality: Product attributes can appear only in images or videos, never in text.
  • Multilingual concerns: Merchants operate globally, using different languages and market terminologies.

This patchwork data leads to classic e-commerce pain points: weak semantic search, poor faceting, duplicate listings, and difficulty surfacing the most relevant products. As shopping shifts toward AI agents and conversational interfaces, the stakes rise—even the best models stumble when the underlying data is inconsistent.

Four Layers of the Global Catalogue

Global Catalogue is a unified foundation for product knowledge that works through four integrated layers, each addressing a distinct problem.

Data Foundation

Commerce data is high-volume and volatile. The foundation layer processes over 10 million product updates daily in a streaming fashion from merchant uploads, APIs, apps, and integrations. A custom schema-evolution system keeps data compatible as merchants change their structures, while change data capture logs every product modification for consistent historical views and incremental processing.

Product Understanding

This layer converts unstructured listings into standardized metadata through a set of coordinated tasks:

  • Product classification: Mapping each product into Shopify’s hierarchical taxonomy.
  • Attribute extraction: Identifying and normalizing features like color, size, material, brand, and model.
  • Image understanding: Pulling color (as hex codes) and visual attributes, plus evaluating image quality.
  • Title standardization: Condensing verbose titles (e.g., “iPhone 16 Pro 256GB Silver” becomes “iPhone 16”).
  • Description analysis: Summarizing long descriptions and surfacing key selling points.
  • Review summarization: Generating quality and sentiment signals from customer feedback.

The design treats this as a multi-task, multi-entity problem. Each entity represents a different grain of the catalogue—media, products and variants are primary examples, but sellers and reviews are also entities. Instead of building a separate model per task, Shopify fine-tunes a single vision language model per entity to handle multiple tasks at once. This is more than an efficiency play; the tasks have interdependencies. Category inference informs text summarization, which in turn can refine the classification. The result is higher-quality output than isolated models would produce.

Shopify’s open-source product taxonomy defines the space of possible categories and attributes. LLMs continuously analyze listing patterns to propose new attribute and category nodes, and changes pass through both automated and human review. Each attribute is linked to a taxonomy node, so the properties relevant to a given product type propagate down the hierarchy as the taxonomy evolves.

Product Matching

When different merchants sell the same physical item, the system needs to detect it. This happens through a multi-stage pipeline that clusters related listings while protecting precision.

Candidate generation uses locality-sensitive hashing, embedding-based clustering, and other probabilistic methods to find fuzzy connections between products. Deterministic matches are also used—universal product codes and high-confidence feature combinations like same title plus same image. Candidate clusters are then run through a cascade of discriminator models that remove questionable edges. That “edge pruning” step matters because a single wrong edge can balloon a cluster with unrelated products.

Matching is modeled as a bipartite graph:

  • Products sit on the left-hand side
  • Attributes—both deterministic features and fuzzier candidates—form the right-hand nodes

Computing connected components over this graph yields product clusters that receive Universal Product IDs.

Reconciliation

For each cluster, reconciliation builds a canonical product record from aggregated metadata:

  • Attribute merging: Complementary data from different listings is unified, building the broadest accurate set of specifications and options.
  • Variant normalization: Color names, sizes, and other variant dimensions are standardized across listings.
  • Content aggregation: Descriptions, specs, and review summaries are merged while selecting the strongest media for the item.

The final canonical record is the authoritative source downstream systems consume.

A Fine-Tuning Strategy for Speed and Cost

Fine-tuning adapts a pre-trained model to a specific task, but commercial APIs become prohibitively expensive at Shopify’s volume. Instead, the team fine-tuned smaller open-source vision LLMs and achieved better performance with more control over costs.

Three models have been deployed in succession: LlaVA 1.5 7B, LLaMA 3.2 11B, and currently Qwen2VL 7B. Each transition improved accuracy while reducing GPU requirements. Emerging models are continuously assessed for the accuracy-to-cost tradeoff.

One key discovery was that predicting every field during fine-tuning hurt the model’s generality. The team switched to selective field extraction: for each training example, only a randomly chosen subset of fields is predicted. One instance may ask for category alone; another for category and title; a third only the standardized description. This teaches the model to flex at inference time without retraining.

The results were measurable in production. Selective field extraction preserved generalization, dropped median latency from 2 seconds to 500 milliseconds, and slashed GPU usage by 40% due to fewer generated tokens. The system now serves more requests on the same hardware, improving cost efficiency and scaling throughput to an estimated 40 million multimodal LLM inferences per day.

Training Data and Evaluation

Building reliable fine-tuned models depends on high-quality training and evaluation data. Shopify built an annotation pipeline that pairs LLM agents with human reviewers. Training datasets are structured around individual catalog entities, with data snapshots taken temporally. For each extraction task — category classification, for instance — multiple LLM agents independently analyze a product and propose labels.

Test samples and train samples flow through different paths:

  • For test samples, human annotators see the agents' suggestions in a custom interface and resolve ambiguities into gold labels. Consensus is established for evaluation purposes.
  • For train samples, an LLM arbitrator — a separate model trained to pick the best agent suggestion or abstain — handles the workload at scale, with human fallback when needed. This trades accuracy and scalability against each other so datasets can be assembled far faster than human-only annotation would allow.

The test set review interface gives annotators full context: product image(s), raw description, and LLM suggestions. Annotators can select, reject, or search for the correct label in the taxonomy tree. Randomization is used deliberately to measure and correct for human annotation bias and to push reviewers toward searching for the right label instead of defaulting to one of the suggested options.

Evaluating multi-task extraction models requires more than standard metrics. Shopify uses three evaluation tracks:

  • Task-specific metrics: Precision and recall at multiple levels of the category hierarchy for classification; accuracy for attribute extraction.
  • LLM judge metrics: For generative fields like standardized titles or descriptions, synthetic judges grade outputs against detailed guidelines.
  • Instruction metrics: The models must honor selective extraction requests, so these measure instruction-following capability.

Instruction metrics capture two behaviors. Field compliance rate tracks how often the model outputs only the fields requested. Field invariance rate measures whether answers for a given field stay consistent when the requested output schema changes. That second metric matters: systems query the model for different field combinations at inference time, and adding fields should not perturb earlier inferences.

Continuous Improvement via Active Learning

Shopify's active learning loop keeps the catalogue models current. Two detection mechanisms feed new training data back into the pipeline:

  • LLM judges flag low-quality inferences in production and queue them for human review and retraining.
  • Token probability distributions on model outputs are analyzed. Samples with low output token probabilities signal uncertainty and re-enter the training pipeline for robustness gains.

Serving Infrastructure at Scale

The inference stack handles about 40 million LLM calls daily — roughly 16 billion tokens per day. Several optimizations make this feasible:

  • Triton inference server orchestrates model serving, batching, and routing across the GPU fleet for multiple surfaces (admin UI, Shop app, APIs, bulk pipelines).
  • Dataflow streaming pipeline uses a Kafka-based architecture to write inferences back to data sinks in real time.
  • FP8 quantization reduces GPU memory footprint without sacrificing accuracy, allowing larger batch sizes.
  • Key value cache stores and reuses previously computed attention patterns.
  • Selective field prompting lets different surfaces request only the fields they need, dropping median latency from 2 seconds to 500ms and cutting GPU token usage by 40%.

Integrated Impact and Roadmap

The catalogue is live across Shopify's ecosystem. In the merchant admin, it offers real-time category and attribute suggestions when products are created. Search and recommendations benefit from enriched data and universal identifiers for better matching and faceting. Embeddings creation — product and variant representations built from standardized output — underpins personalized ranking and recommendations.

Downstream effects already show up across product surfaces:

  • Search: Canonicalized metadata means broad queries like "best local coffee in Singapore" return relevant long-tail products, not just top brands.
  • Personalization: A unified catalogue lets recommendation systems combine user interaction data with standardized attributes.
  • Conversational commerce: AI assistants (Shopify Sidekick, Shop app chat) use catalog data as structured context for dynamic, needs-based shopping flows.
  • Multi-channel integration: A common schema and universal product IDs make the ecosystem interoperable across channels and partners. The open-source taxonomy and standards extend that compatibility outward.

The engineering work is far from done. Key challenges remain on the near horizon:

  • Balancing scalability, accuracy, and latency: Quality must hold up against the pressure of speed and cost at current traffic volumes.
  • Unified versus multi-model: Each key entity currently has a dedicated, size-optimized model. Shopify is exploring a consolidated, multi-entity model and expects similar gains from tackling multiple tasks simultaneously.
  • Graph-based reasoning: Using LLM reasoning over entity relationship graphs is a promising direction for entity resolution.
  • Continuous pipeline improvement: Active learning, dynamic retraining, and infrastructure scaling remain ongoing priorities as data and requirements evolve.