Data-Centric Machine Learning: Building Shopify Inbox’s Message Classification Model - Shopify

Why Training Data Quality Beats Model Tuning

Building Shopify Inbox meant understanding what buyers were actually chatting about—product recommendations, shipping questions, or simple greetings—so merchants could prioritize replies that convert. With millions of messages flowing through the app monthly, that’s a serious NLP problem. But solving it didn’t require inventing a new model architecture. Off-the-shelf pre-trained models are readily available; the real challenge was getting the training data right.

Our first prototype used a pre-trained Universal Sentence Encoder from TensorFlow Hub, which outputs whole-sentence embeddings sensitive to word order—important for distinguishing “Love! More please. Don’t stop baking these cookies.” from “Please stop baking more cookies! Don’t love these.” To build training data quickly, we clustered messages by semantic similarity using UMAP and HDBScan. After manually assigning topics to about 20 clusters, we hand-labeled a few seed messages per topic and used the embeddings to surface similar examples—e.g., from “Can you help me order?” we found “How to order?” and “How can I get my orders?”

Training a simple model—an embedding layer plus two dense layers and a logits output—gave us roughly 70 percent accuracy, and it could classify only 35 percent of messages with high confidence. The prototype was useful: it revealed that order status questions dominate chat traffic, informing the Instant Answers FAQ feature. But the labels themselves were often unreliable. Our scrappy embedding-based labeling approach wasn’t producing ground truth.

To improve accuracy, coverage, and speed, we shifted focus from the model to the data—revisiting labels and manually annotating more messages in a systematic way.

Rebuilding the Taxonomy From Scratch

Examining our prototype’s message clusters exposed serious problems: broad topics conflated distinct meanings—shipping availability questions (pre-purchase) were lumped together with order status queries (post-purchase). Some topics had few examples; many messages fit no topic at all. Training on such an unbalanced, noisy dataset couldn’t yield high accuracy or coverage.

We needed a labeling system that was accurate, useful for merchants, and unambiguous enough for annotators to apply consistently. Collaborating with our staff content designer, product researcher, and part-time support advisors—people who understand Shopify merchants and their buyers—we spent two months sifting through hundreds of messages. The result was a taxonomy documented in a shared spreadsheet: each topic listed with a description, cross-references, disambiguations, and sample messages. That document became the project’s source of truth for data scientists, engineers, and annotators.

In parallel, we evaluated newer pre-trained models. The Transformer family was already powering our product categorization model, so we chose DistilBERT for its balance of performance, resource usage, and accuracy. Early prototyping on a small dataset built from our nascent taxonomy showed clear improvement over version 1.0, reinforcing our bet on data quality.

The final taxonomy spanned more than 40 topics under five categories: Products, Pre-Purchase, Post-Purchase, Store, and Miscellaneous. The hierarchy mirrors how a buyer’s message might be approached: where is this shopper in their journey? Is the question about product details like color or size, about payment methods, or about a broken item and refund? Each category includes an other topic for messages too vague to fit a specific label. We excluded those examples from training—since we couldn’t classify them ourselves, they wouldn’t teach the model much—and instead set probability thresholds in production to decide when to ignore low-confidence predictions.

Aligning Annotators Through Training

With a taxonomy that large, consistency across annotators was critical. We ran several training sessions describing the classification project and philosophy, then split annotators into two groups to classify the same set of messages. That exercise served two purposes:

1. It familiarized the team with our in-house annotation tool.
2. It let us measure inter-annotator agreement.

Multiple rounds of these exercises were time-consuming but worthwhile. They exposed inconsistencies that led us to refine the taxonomy—eliminating ambiguous descriptions, adding examples, and adding or removing topics. They also confirmed that annotators were genuinely aligned on how to classify messages.

Annotation Goes Live

Once the team was ready, annotation began in earnest. We set up a Slack channel for the annotators to collaborate on tricky messages in real time, making the reasoning behind classifications visible to everyone. Preprocessing stripped single-character messages and emoji-only content. During annotation, we also excluded noise like buyers pasting full email bodies from Shopify store confirmations—not real chat messages—and, given scope and resource constraints, set aside non-English messages. In short: garbage in, garbage out.

De-identifying Message Data

Buyer messages occasionally contain personal information (PI) such as email addresses or phone numbers. To protect merchant data, the team flagged messages containing PI and replaced those values with mock data before training.

Annotators first identified messages with PI. An open-source library called Presidio then analyzed and transformed the flagged content. Crucially, Presidio ran inside Shopify’s data warehouse, so merchant data never left the company’s systems. Presidio recognizes many PI types, and its anonymizer offers multiple operators: complete removal, partial masking, or substitution.

For substitution, the team paired Presidio with Faker, a customizable, localized library whose providers generate realistic addresses, names, locations and URLs. The fabricated example below shows Presidio running over a sample message:

Automation didn’t catch everything. The team inspected before-and-after output for any surviving PI and manually replaced leftovers with placeholders like <PERSON>. A final script swapped those placeholders for Faker-generated values.

Correcting for Class Imbalance

As the annotation campaign progressed, a persistent trend emerged: certain taxonomy topics were heavily overrepresented. Buyers, it turns out, ask a lot about products. With thousands of messages already labeled, re-splitting the popular topics for reclassification wasn’t feasible — yet the model still needed solid coverage across minority classes.

The solution leveraged the team’s continuous training loop. A model was retrained on labeled data as it became available, and its predictions were used to filter the pool of unlabeled messages. By excluding anything the model confidently classified under the overrepresented topics, the remaining candidates were concentrated among underrepresented topics or cases where the model was uncertain. Annotators then manually labeled those messages.

Outcome and Ongoing Work

The result of the dataset effort was a measurable improvement over the first prototype. The final model achieved its targets for higher accuracy and coverage:

Part of that success came from collaboration with subject matter experts beyond the data science team. Support advisors, a staff content designer, and a product researcher contributed perspectives that the data scientists alone couldn’t supply.

The project remains a work in progress. Conversation topics in Shopify Inbox shift as trends and sentiments change, so the taxonomy, training data, and models all require continued updates to keep standards high.