Teaching machines to see products the way shoppers do
Two merchants can sell the same protein powder and structure their catalogs completely differently. One creates a single listing with flavor variants; another creates a separate listing per flavor. Both are correct on their own storefronts, but when an AI shopping agent searches across Shopify's billions of listings, it has to reconcile those differences without being told.
Shopify Catalog is a unified intelligence layer that standardizes product data across the platform and exposes it to developers and agents through the Catalog API. At its core is product clustering: grouping related variants and products under a single Universal Product Identifier (UPI), regardless of how each merchant structured their own catalog.
Getting this wrong breaks results in two ways. Precision failures merge distinct products — men's and women's versions of the same sneaker, for instance — so a buyer can accidentally receive the wrong item. Recall failures leave a variant out of a cluster, so a shopper searching for a specific jacket color either finds nothing or sees a confusing duplicate listing.
The two metrics pull against each other: tightening thresholds boosts precision but misses valid matches; loosening them improves recall but merges things that shouldn't be together. Shopify chose precision-first. If precision slips, the wrong product surfaces; if recall slips, the item is still findable, just ungrouped. Buyers can forgive a missing variant more easily than a wrong product. Teams measure both against curated ground-truth datasets with human-labeled samples across categories and merchant types, plus LLM-based judges calibrated against human annotations, so quality can be tracked as the catalog changes daily.
One store at a time
The full global matching problem — comparing every product against every other product across millions of merchants — has an impossible candidate space. Shopify started with a simpler observation: many merchants sell products unique to them, with their own brands and designs. For those shops, the question isn't which products across Shopify are identical, but which listings inside one shop are variants of the same product.
That partitions the work into millions of independent, store-level clustering tasks. It also builds intuition about what product identity means before tackling cross-store matching.
Why LLMs earned the job
A human browsing a shop typically doesn't need a formal ontology to group listings. They read titles, descriptions, tags, and option names in context. They can tell that "Geometric Design Black Flatwoven Rug" and "Geometric Design Cream Flatwoven Rug" are the same rug in different colors — or that Midnight Blue and Sage Green paint are distinct products because color is what the buyer is choosing there.
That judgment relies on unstructured context, and merchants can put the same signal in different fields. LLMs extract those signals and standardize them into a consistent structure. But prompting a model to "just cluster these products" produces inconsistent output. The prompt needed a principle.
The core value proposition test
Shopify's framework asks one question: what is the buyer primarily purchasing this product for? If an attribute doesn't change the answer, it's a variant. If it does, it's part of the product identity and splits into a separate UPI.
For protein powder, flavor doesn't change the core value proposition — the buyer wants nutrition and protein content, so chocolate and vanilla are variants. For paint, color often is the core value proposition, so Midnight Blue and Sage Green are separate products.
The prompt embeds this directly. First, the model analyzes the shop's naming patterns: which terms repeat, which vary, which define product families. Then it extracts the brand, using the merchant's vendor field as the primary signal. Finally, it applies the core value proposition test to identify model-defining attributes. Teaching examples help the model learn the principle and apply it to categories and naming conventions it has never encountered.
Rules first, LLMs where it counts
Not every product needs an LLM. A pre-filtering step called the singleton detector parses each merchant's storefront theme code to determine whether clustering is needed at all. Many Liquid templates define explicit cross-product linking patterns — metafield references, tag-based grouping, collection lookups. If a shop's products are already standalone with no variant grouping, each gets its own UPI without touching the clustering pipeline.
Only a small percentage of shops actually require LLM-based clustering. The rest resolve deterministically through merchant-defined schemas, tag matching, title similarity, or metafield rules. Layering deterministic rules first and reserving LLMs for genuinely ambiguous cases reduces cost and avoids precision-hurting false merges.
Two-stage LLM pipeline
Context in neighborhoods, not haystacks
Most merchants are too large for a single prompt — a few list millions of products, and even a mid-sized catalog of a few thousand items exceeds what an LLM can handle well in one pass. But clustering depends on context: feeding the model isolated products loses the naming patterns and category cues that reveal a shop's structure.
The smallest useful unit of context turned out to be neighborhoods: sets of products close enough to expose naming conventions, diverse enough to show what varies within a line. Pre-chunking assembles those neighborhoods before involving the LLM.
ANN retrieval plus average linkage
An initial approach using linkage-tree clustering on product features worked for small shops but degraded beyond roughly 10,000 products. The replacement combines approximate nearest neighbor (ANN) retrieval with a modified average-linkage method:
- Build a nearest-neighbor graph: every product is embedded, then HNSW via FAISS connects each to its 100 closest neighbors by cosine similarity.
- Sparse average linkage: starting from singletons, the closest cluster pairs merge using UPGMA — an unweighted average of member-to-member distances. Pairs not connected in the graph get a penalty just above the threshold, preventing weak jumps across the graph. Merging stops when the closest remaining pair exceeds the distance threshold (0.25) or when a cluster reaches the maximum chunk size of 200 products.
The output: semantically related chunks the LLM can read as a coherent neighborhood — similar enough to surface shop-level conventions, not so tight that everything is a duplicate. From there, the model applies the core value proposition framework to decide which listings belong under a single UPI.
Propose, Then Check: A Two-Pass Clustering Pipeline
Once coherent chunks are assembled, Shopify runs a deliberately simple two-stage process. The first stage proposes clusters; the second critiques them.
Stage 1 extracts brand and model strings for each product in a chunk and assigns universal product identifiers (UPIs) accordingly. Stage 2 scans each proposed cluster for mismatches and flags outliers. The default bias in the second stage is to keep items together unless there is clear evidence that the core value differs.
Stage 1: Extracting Brand and Model
Each chunk is sent to the LLM with shop context (name, domain, brand) plus per-product metadata: titles, URLs, and additional fields. The model must return a strict JSON object that assigns a brand and model string to every product ID. Products sharing the same brand:model pair are grouped under one UPI, scoped by shop as shop_id:brand:model.
The system prompt encourages the model to first output a patterns array, describing the naming conventions it observes in the shop — how names vary and which terms look model-defining — before labeling any individual product. This pushes the model to reason about the shop’s language rather than pattern-matching titles in isolation.
Stage 2: Detecting Outliers
Creating clusters from scratch is a form of creative work: deciding how many groups exist, what they are, and where every product belongs. The possibility space is enormous, and a single pass must be globally consistent. Critiquing a proposed cluster, by contrast, is a narrow judgment call. Given a cluster of sweatshirt blazers, noticing that three tailored blazers do not fit only requires local judgment — not a full re-clustering of the shop.
The second pass takes all products in a proposed cluster (titles and additional metadata) and returns a reason string plus an outliers array of product IDs that do not belong. The default is to keep items together unless the core value is demonstrably different.
Because critique is cheaper and more reliable than creation, this stage catches mixed product lines, accidental merges (such as licensed collaborations pulled into a baseline) and bundles mixed with standalone items, all without re-solving the entire clustering problem.
Making LLMs Reliable: Dynamic Structured Outputs
“Here are 200 products — group them” sounds straightforward, but free-form prompts failed in practice. The model would skip items, merge IDs, invent brands absent from the data, or return malformed JSON. Reliable clustering requires exactly one brand and model for every product ID in a chunk. A missed ID never receives a UPI; a hallucinated brand splits products that belong together.
Early attempts used regex parsers, retry loops for bad JSON, and heuristics to catch missing items. The approach was brittle, costly, and error-prone.
The turning point was adopting OpenAI Structured Output with strict JSON schema enforcement. Rather than relying on the LLM to produce well-formed output, the system defines a schema the model must conform to. The innovation is that the schema is generated dynamically per chunk: every product ID in the input becomes a required property in the output products object, with brand and model string fields. The schema makes it impossible for the LLM to return a response that skips a product.
That guarantee made chunking viable. A shop of any size can be split into arbitrary chunks, and every product is classified regardless of chunk boundaries. Without it, the system would need complex reconciliation logic to handle products the LLM forgot to mention.
A later iteration improved efficiency by remapping product_id values. Instead of sending shop-specific IDs like [product_id: 12345, product_id: 12346, …], each prompt normalizes them to [product_id: 1, product_id: 2, …]. Prompts become more reusable across runs while preserving the ability to map results back to original products. This raised cache hit rates and cut costs.
Schema as a Design Tool
The schema also shapes model behavior more reliably than prompt text alone.
- Ordering controls reasoning. The schema requires the patterns array before the products object, forcing the LLM to analyze naming patterns before extracting brand and model values. Field order in the schema measurably changed extraction quality.
- Enum constraints prevent hallucination. In stage 2, the outliers array uses an enum constraint listing every valid product ID in the cluster (capped at 500). The LLM can only output IDs that actually exist — no invented IDs, no typos, no
"product_123"when the real ID is"7891234567890". - Schema as hyperparameter. The output schema influences cluster quality as much as the prompt. Cleaning non-ASCII tokens from the output structure improved recall by 8% on Toloka evaluation data. Misaligning title format between prompt input and schema expectations lowered recall on the GTX dataset. Removing all structure produced surprisingly few parsing errors (0.25%) but did not improve quality. Structured output won on both reliability and quality.
From Intra-Store to Cross-Store
The team started with a precision-first stance, defining product identity as the core value proposition. The intra-store pipeline that emerged scales by layering judgment: rules before heuristics via a singleton detector, pre-chunking with ANN and sparse average linkage to give the model the right context, and a two-stage LLM flow that proposes and then critiques.
Dynamic structured outputs turned reliability into a hard engineering constraint rather than a hope, guaranteeing every product is labeled and making chunking feasible at any shop size. Together these pieces cluster billions of products with high precision while recall is pushed upward, measured against continuously refreshed ground truth and calibrated LLM judges.
But buyers do not shop in a single store, and agents will not either. The next step is cross-store clustering: unifying identical products across merchants under a single, global UPI so agents and developers can search, compare, and recommend with true catalog awareness. That effort moves intra-store judgments into a much noisier world.
This article contains contributions from Boris Nazarov, Ilia Kuchumov, and Andrei Danilchenko.
