Data at Scale Creates a Navigation Problem

The growth of data is relentless. The global datasphere was estimated at 33 zettabytes in 2018 and is projected to reach 175 ZBs by 2025. As organizations accumulate more data, the problem shifts from storage to sense-making: data assets are constantly aggregated into tables, reports, and dashboards to solve specific problems, feed machine learning models, or serve as inputs to other assets. This process repeats, producing a sprawling web of interconnected assets.

At Shopify, we built a tool called Artifact to tackle the data discovery and management challenges that come with this scale. Our goal was to make it easier for data professionals to find relevant assets, understand their context, and manage their impact.

Three Core Challenges

We categorized our data discovery issues into three distinct areas: curation, governance, and accessibility.

Curation: Finding What Already Exists

The fundamental question our teams faced was, “Is there an existing data asset I can utilize to solve my problem?” Prior to Artifact, answering that meant asking colleagues in person, messaging on Slack, digging through code on GitHub, or sifting through job logs. This scattered approach led to multiple sources of truth, lost context, and duplicated effort.

When we surveyed our Data team, 80% felt the old discovery process hindered their ability to deliver results. After Artifact launched, that sentiment dropped to 41%.

Governance: Managing Change and Ownership

The second challenge centered on impact analysis: “Who is going to be impacted by the changes I am making to this data asset?” Shopify’s data teams lacked granular ownership information and standardized change management processes. With each team practicing its own approach, revisions and updates were hard to track, leading to a loss of context for anyone using new or unfamiliar assets.

Initially, only 16% of the Data team felt they understood how their changes impacted other teams. With Artifact’s help, that figure rose to 46% as teams gained clearer visibility into their downstream consumers.

Accessibility: Surfacing Insights to Stakeholders

The final challenge was making relevant, well-documented data points available to stakeholders for decision-making. Insights derived from reports and dashboards often got lost in Slack conversations or archived presentations. The lack of metadata around these insights directly impacted decision-making, increased duplication of effort, and created a dependency on a data-as-a-service model that limited our team’s scalability.

Building Artifact

Through extensive conversations with data teams and stakeholders, two key themes emerged:

  • Data assets and their associated metadata form the context that drives discovery.
  • Discovery is iterative, with many possible starting points.

These insights guided the design of Artifact, a search and browse tool built on a centralized metadata model. It aggregates information from various data processes, allowing teams to discover assets along with their documentation, lineage, usage, and ownership. This context enables teams to leverage data more effectively in their roles.

The User Experience

Artifact’s landing page offers two entry points: browsing assets by team, source, or type, or performing a plain-English search. The default view lists all assets ordered by usage, which helps users who aren’t sure what they’re looking for build context before refining their search. Usage and ownership information are displayed up front, highlighting well-adopted assets and providing a path for further exploration.

Search results are powered by Elasticsearch, which indexes titles, documentation, schemas, and descriptions. Each result gives enough detail—name, description, ownership, and total usage—for users to decide whether to dive deeper. Clicking through leads to a details page with metadata organized across horizontal tabs and a sticky vertical navigation bar.

Lineage information is particularly valuable. Powered by a graph database, the lineage view allows users to search and filter dependencies by source, direction (upstream vs. downstream), and distance (direct vs. indirect). This feature provides context on how assets are utilized by other teams and alerts owners to potential downstream impacts of their changes.

The Architecture

Before starting development, we established three guiding principles:

  1. Solve the biggest user obstacles with the simplest possible solutions.
  2. Design generically to ease future integrations and limit technical debt.
  3. Iterate quickly to fail small and learn clearly.

With these in mind, we built a generic data model and a simple metadata ingestion pipeline that pulls information from various data stores across Shopify. The metadata extractor also constructs the dependency graph for lineage. After processing, everything is stored in Elasticsearch indexes, and GraphQL APIs serve the data to the UI via an Apollo client.

Trade-offs We Made

We made several deliberate decisions during the build process that warrant explanation.

Build Versus Buy

We evaluated enterprise and open-source cataloguing tools but found common limitations:

  • Every organization’s data stack is different; downstream business context creates requirements that cannot be satisfied by a one-size-fits-all solution.
  • Existing tools captured only a partial view of data discovery, typically missing usage information, communication, and change management metadata.
  • Shopify’s wide range of data assets demands adaptable metadata models and interactions that market tools don’t support without heavy customization.

Given those findings, buying would have meant significant customization, technical debt, and incremental integration effort. Choosing to build provided the best fit, the most flexibility, and full control over our technical debt footprint.

Metadata Push Versus Pull

We originally hoped that each data tool would expose a metadata API for us to consume. While the teams responsible for those tools understood the value, building new APIs didn’t fit their roadmaps. Speed to market was a priority, so we opted for jobs that pull metadata from their processes. This was an acceptable workaround, with a clear future path toward API-based integrations.

Scoping the Initial Data Assets

Shopify’s pipelines generate a wide range of assets—tables, streams, reports, notebooks, algorithms, dashboards, and more—that we couldn’t support all at once. We prioritized the first release based on these criteria:

  • Frequency of use: how often assets appear across data processes.
  • Impact to end users: the immediate value to users and stakeholders.
  • Ease of integration: the effort needed to bring an asset into Artifact.

Top queryable data assets, along with their downstream reports and dashboards, made the initial cut. This gave users the highest impact for the least build time, with the remaining asset types prioritized and added to our roadmap for subsequent releases.

Adoption and Next Steps for Artifact

Since its release in early 2020, Artifact has seen strong uptake across both data and non-data teams at Shopify. Current usage exceeds initial projections: more than 30% of the Data team interacts with the tool on a weekly basis, and the monthly retention rate stands above 50%. These figures outperform the original targets of 20% weekly usage and 33% monthly retention.

The immediate roadmap centers on adding high-impact data assets that were excluded from the first release and integrating with newer data platform tooling. Looking further ahead, the team plans to address data asset stewardship, change management, notification services, and expose APIs so other teams can consume metadata. The long-term goal is for Artifact to supply the data context every Shopify team needs for better decision-making across the organization.

Ultimately, Artifact is intended to function as a structured toolbox for Shopify’s teams—boosting productivity, lessening business owners’ reliance on the Data team, and broadening access to data across the company.