Bringing Analytics Out of the Reports Section
Shopify merchants have relied on the Analytics section of the admin since 2013 to track performance and guide decisions like fulfillment priorities and acquisition strategies. While that section, with its Overview Dashboard, customizable Reports, and Live View, remains a powerful destination for deep dives, we spotted a gap in the merchant workflow. User research and quantitative analysis showed merchants frequently toggled between the Analytics section and other admin pages to make data-informed decisions.
To close that gap, we launched an in-context analytics experience directly on the Orders page—the most visited page in the admin. The goal was to surface real-time and historical metrics at the moment merchants are managing orders, giving them immediate insight into the health of their workflows without leaving the task at hand.
Defining Metrics Around Merchant Goals
The first step was identifying which metrics to surface. Working with product managers, we mapped common merchant workflows on the Orders page (like fulfillment) to the underlying goals those workflows serve (such as reducing fulfillment time). Comparing those goals against available data, we landed on three categories of information:
- Top-level metrics (e.g., median fulfillment time).
- Dimensions for filtering those metrics (e.g., location).
- Complementary visualizations (e.g., a distribution of event durations) to add context to the headline number.
For each metric, we ran through specific use cases to test whether seeing the value would trigger a useful action. A merchant noticing a higher median fulfillment time this month versus last should be able to drill into the time series to look for a trend. This exercise confirmed we were proposing metrics that would give merchants actionable signals, not just pretty charts.
Testing Against the Shape of Real Data
Shopify supports millions of merchants at different stages, so we needed to verify our proposed analytics would work across that spectrum. We segmented stores by order volume, length of operation, fulfillment and delivery methods, geography, and industry. For each segment, we asked a standard set of questions about every candidate metric: Would it be useful? For which segments? How volatile was it over time? Over what time window did it provide the most value?
| What we saw | Action we took |
| We don’t have the data necessary to compute a metric, or a metric is always 0 for a merchant segment | Only show the metric to the stores where the metric is applicable |
| A metric stays constant over time | The metric isn’t a sensitive enough health indicator to show in the Orders page |
| A metric is most useful for longer time periods | The metric will only be available in reports where a merchant could look at those longer time periods |
Analyzing real store data for different segments often pushed us to refine the product. Some metrics only made sense for particular merchant types, such as those using shipping carriers instead of local delivery. In other cases, the data’s variability—like extreme outliers in the distribution of an event’s duration—made us gravitate toward visualizations, like histograms, that better conveyed the full picture. These explorations showed us we always had to validate proposed experiences against the data to ensure they would truly fit the needs of all merchants.
Because we accumulated hundreds of data points across these investigations, we built a slide deck with one finding per slide, noting its implications and recommended product decisions. This structure kept a consistent, digestible format for the broader team.
Prototyping with Production Data
With the metrics selected and validated, we looped in the UX team. Rather than using mock or synthetic numbers, we prototyped visualizations against real data from the merchant segments we had identified.
|
|
That workflow produced immediate feedback. When building out a report on event durations by location, we noticed a significant spread in values. That drove the decision to visualize the data as a histogram of the event-duration distribution—far more informative for merchants than just showing averages in a table. By mocking up graphs natively in our data science tools, we delivered working examples of visualizations that engineering and UX could then refine into the final product.
Creating a Single Source of Truth for Every Query
We treated every metric surfaced on the Orders page and in the associated reports as a uniquely defined query. To ensure consistency across the Web, Android, and iOS clients, we needed one definitive reference for the entire experience. That reference took the form of a data specification sheet that for each metric defined three things:
- The merchant goal being served.
- Qualification logic to ensure a metric only appeared if it was relevant to a given merchant.
- The final query, including the exact data source, fields, and aggregations used.
This approach meant any developer or product manager could inspect store data or understand a metric while building, without having to guess at underlying definitions.
Model Design and the Streaming Trade-Off
After the specification was set, we moved to dataset design. We defined the business processes the model had to support (fulfilling an order, marking it in-transit, and marking it delivered) and chose an atomic grain—one row per fulfillment, in-transit, or delivery event. That level of granularity gives the model flexibility to support future dimensions and products. We finalized the schema with the needed dimensions for the experience.
Even though the product required real-time metrics, we deliberately built a batch pipeline first instead of a streaming one. There were three concrete reasons for this sequence. First, building the batch model in our standard environment let us iterate on cleaning logic and thoroughly sanity-check the results before it moved to the production analytics service. Second, since we had existing familiarity with batch tooling, we could stand up the model quickly and let engineers begin using its data as a placeholder while they built the experience. Third—and critically—the batch model gave us a way to backfill full history. Productionizing real-time data with a lambda architecture made it possible to serve historical records from the batch model and only process the most recent tail of new events in the streaming pipeline. That approach minimized any extra burden on streaming infrastructure while still meeting the requirement to show near-current numbers.
Measuring the Impact of In-Context Analytics
Shopify evaluates projects against a simple question: what merchant problem are we solving? To answer that rigorously, the data science team uses a measurement framework that forces them to think beyond raw adoption numbers. The framework covers the problem statement, the expected outcome, the signals that indicate success, the factors that could move those signals (both positive and negative), a baseline for comparison, and any relevant audience segments.
For this project, the framework produced a concrete example. The problem: merchants shouldn't have to leave their operational workflow to find decision-making data. The hypothesis: putting analytics directly in the Orders page would reduce the need to jump between pages. The success signal: merchants use the new data while completing tasks on the Orders page, with a measurable drop in navigation to the separate Analytics section. The team also anticipated a caveat: even a valuable new feature might see low early adoption, not because the data is irrelevant, but because merchants haven't discovered it yet.
The baseline was simple—compare page transitions between Orders and Analytics before and after the release. Context also matters: segmenting by business size could reveal that larger merchants, who often delegate order fulfillment to staff, may have no interest in analyzing that data themselves.
Because this project served multiple audiences with different goals, the team resisted merging everything into a single success dashboard. Instead, they built separate dashboards for each goal and audience, documenting what questions each one was meant to answer. Data engineers, for example, tracked the performance of the new data model independently of the merchant-facing experience.
What Merchants See Now
The shipped feature surfaces a set of key metrics directly on the Orders page. Merchants can view total orders (overall and by time period), the number of ordered items, returned items, fulfilled orders, delivered orders, and the median fulfillment time. Each metric covers the current day, the last seven days, and the last thirty days, with a green or red indicator showing whether the value has improved or worsened compared to the previous equivalent period.
Clicking any metric opens a dedicated report for deeper analysis:
- Orders Over Time Report: Total orders received in the selected window, plus average units per transaction, average order value, and returned items.
- Product Orders and Returns Report: Identifies best sellers and the products most frequently returned.
- Fulfillment, Shipping, and Delivery Times Report: Tracks an order's full journey from receipt to customer delivery.
- Fulfillments Over Time Report: Shows orders fulfilled, shipped, or delivered over the selected period.
Lessons for Future Analytics Projects
The project produced several takeaways the team will carry forward. First, tight collaboration across disciplines, supported by central documents like data specification sheets and highlight decks, kept everyone aligned during the build. Second, prototyping with real, segmented data proved essential—merchants vary widely in size and operational complexity, so a design validated on one segment may not work for another. Third, modelling the data models in the batch environment before switching to streaming significantly reduced risk and unblocked the engineering team.
The work isn't finished. The team is still validating that the feature has no unintended negative impacts on users. Going forward, Shopify plans to extend the same in-context approach to other surfaces beyond the Orders page, putting data where merchants already work rather than forcing them to go looking for it.
