Why Search Evaluation Is Hard
Shopify's Help Center processes searches against a library of thousands of articles. A query like “shipping” can return hundreds of results, and the ranking algorithm decides which ones the user sees first. The current production algorithm, Vanilla Pagerank, boosts articles by their total view count across all searches. That approach has a weakness: it can surface popular articles that aren't actually relevant to the specific query. For example, a search for “delete discounts” might return articles heavy on the words “delete” or “discounts” but miss the article on deleting discounts specifically.
To improve on that, the team built a new algorithm, Query-specific Pagerank. Instead of using global popularity, it boosts results based on click frequency from historic searches that contain the same search term. In other words, it favors the articles that users most often clicked on for similar queries.
The core problem with any search change is that it can help some queries while hurting others. To build confidence in the new algorithm, the Data Science & Engineering team relies on a three-step evaluation framework: collect data, run offline evaluation, then validate with online A/B tests.
Step 1: Collecting Ground Truth Data
Before evaluating an algorithm, you need a labeled dataset that tells you which articles are relevant for which user intents. For the Help Center, that data comes from two sources: Kafka events and human annotation.
Events from Kafka
Users generate a stream of search interactions: queries entered, results clicked, and feedback left. The team collects these events in schematized streams using Kafka, then models them in an ETL pipeline. The output is a search fact table that aggregates details about each search session based on the behaviors being analyzed.
Kafka-based data enables three capabilities:
- Monitoring product changes for adverse impacts on user experience in near real time.
- Real-time A/B test assignment, where some users are served results from algorithm A and others from algorithm B.
- Streaming interaction ingestion to provide immediate feedback to the search product.
Human Annotation
Labeled datasets give you a “ground truth” against which to measure an algorithm. Within Shopify, the Support team annotates search results, using their expertise to judge relevance. Manual annotation yields high-quality, trustworthy labels, but it is slow and expensive. Automated alternatives like click models exist, but for the Help Center, human judgments are preferred because of the Support team's domain expertise.
The annotation process itself is straightforward:
- A query is paired with a document or a result set that might be a relevant match.
- An annotator assigns a relevance rating to the query-document pair.
- Labels are combined with inputs to produce the final labeled dataset.
There are a few different ways to structure the annotations:
- Binary ratings: The annotator answers “Is this document relevant?” with a 1 or 0.
- Scale ratings: The annotator rates relevance on a 1-to-5 scale, where 4 and 5 count as a hit and anything lower is a miss. This gives interval data that can be converted into binary categories.
- Ranked lists: Given a query and a set of documents, the annotator ranks them from most to least relevant.
The annotation type you choose depends on the evaluation measures you plan to use. For the Help Center, the team used scale ratings with a worded list—bad, ok, good, great—to make the judging task clearer. Those words are then converted to numeric labels for computing performance metrics.
One caveat: datasets become stale. The Help Center's article collection changes frequently, so annotation projects need to be re-run regularly, or augmented with unseen data points for training.
Step 2: Evaluating Offline Metrics
Offline evaluation lets the team measure whether Query-specific Pagerank improves ranking effectiveness without risking user experience. It works by replaying thousands of historical queries through both the current and proposed algorithms, then scoring results against the curated relevance labels. Each run compares two key measures: Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (NDCG).
Mean Average Precision
MAP measures relevance among the top N results. Since users rarely scroll through hundreds of results, looking at only the first N is a practical proxy for what users actually see. Precision@N is the fraction of relevant results among those top N items; MAP is the average of those precision scores across all queries in the test set. The score penalizes algorithms that return irrelevant documents before relevant ones.
A limitation of MAP is that it requires a binary relevance cutoff. If a query-document pair is rated ok, good, or great, you might classify it as relevant — meaning the difference between ok and great gets ignored entirely.
Normalized Discounted Cumulative Gain
DCG addresses that flaw by keeping the non-binary grade while applying a logarithmic discount as results move down the ranking. This ensures a highly relevant article appears in position 1 counts for more than the same article buried at position 8.
DCG has a drawback, though: queries that return more documents naturally produce higher scores, regardless of ranking quality. NDCG solves this by dividing the raw DCG by the ideal DCG — the score you'd get from perfectly sorting results by relevance. The result is a normalized score between 0 and 1.
Comparing these scores works well when the differences are clear. But if your old and new algorithms get similar results, the metric alone won't tell you why. To gather deeper insight for future iterations, examine the composition of queries in your test set:
- Frequency: How often does each algorithm return results that are worse than the labeled annotations?
- Velocity: How far off is the ranking position compared to the ground truth?
- Commonalities: Look for queries that consistently perform better under one algorithm, and find what those queries have in common to understand the algorithm's limits.
When the team put Query-specific Pagerank through offline evaluation with MAP and NDCG, the new algorithm returned higher-graded documents more often and posted slightly better scores on both metrics.
Measuring Real-World Behaviour
Online metrics complement offline evaluation by using search logs to observe how real users interact with the product. They are commonly used to judge A/B tests.
The right online metrics depend on your product’s goals. For the Shopify Help Center, whose purpose is connecting users with relevant support articles, the meaningful signals are interaction with results, how far users scroll to find what they need, and whether they still end up contacting Support.
Before running an A/B test, you have to define those measures and set expectations for how the new algorithm should shift them. Four metrics are particularly useful in this evaluation:
- Click-through rate (CTR): The share of users who click a surfaced result. Great results get clicked, so target a high CTR.
- Average rank: The mean rank of clicked results. Aims for the most relevant articles first, so a low average rank is good.
- Abandonment: When a searching user doesn’t interact with results, contact support, or return. Some is normal (bots and spam hit search too); it should stay moderately low.
- Deflection: Success when users solve their own problem and don't contact support. High deflection is the target, but it’s nuanced — sometimes contacting support genuinely is the best outcome.
The Kafka data collected earlier feeds these metrics, allowing tracking over time, across user segments, and by search topic. The team, for example, compares CTR and deflection for users in different languages. A/B tests assign users to different algorithm versions to see if a new challenger significantly beats the incumbent.
Such a test on the Help Center works like any other experiment: visitors are assigned to a group that determines which algorithm powers their searches. Enough traffic accumulates, and the metrics reveal which variant wins — say, one with a distinctly higher CTR.
A Test Run: Pagerank Versus Query-Specific Pagerank
The team ran an online A/B test pitting Query-specific Pagerank against the existing Vanilla Pagerank, with users split evenly between the two. Users powered by the new algorithm were:
- Less likely to click past the first page of results
- Less likely to follow up with another search
- More likely to click results in general
- More likely to click the first result shown
- More likely to have a lower average rank of clicked results
In short, the Query-specific group found helpful articles with less effort.
What the Numbers Decided
The evaluation framework showed the new algorithm outperformed the old one on the product’s own terms. The experiment was deemed a success, and Vanilla Pagerank was retired in favour of the Query-specific variant.
The framework has value even when the experiment fails. A drop in performance observed offline or online might be traceable to a subset of queries or particular user segments who fare better under the old behaviour. Either way, the deeper analysis and the documentation you produce feed future iterations and are worth keeping.
Takeaways for Search Evaluation
An algorithm sets the tone for the entire search experience, and evaluating it properly protects your users and your decisions. A framework for that evaluation should stand on three legs:
- Quality labelled data is the baseline for unbiased evaluation — without it, no metric can be trusted.
- Online metrics reveal actual user behaviour, although they are resource-intensive and riskier to deploy.
- Offline metrics allow fast iteration on new algorithms before risking their roll-out in production.
