From repository to discovery engine
Spotify’s internal data tooling has evolved a lot since Lexikon first launched in 2017. Built as a response to an explosion of BigQuery datasets and a surge in hiring across the insights community, Lexikon aimed to centralize access to datasets, research, and analysis. Adoption was strong early on, but the product wasn’t solving a key problem: data scientists were still spending too much time hunting for the right tables.
Survey feedback revealed that discovery remained a top pain point, even though roughly 75% of data scientists used Lexikon regularly. Given that a typical data scientist works with 25–30 datasets per month, slow discovery directly delayed insight production. The team set out to rework Lexikon around three ideas: understanding user intent, enabling knowledge exchange through people, and helping users get comfortable with datasets once found.
Designing for different discovery modes
User research showed that data scientists approach discovery with very different levels of intent. Low-intent discovery happens when someone has a broad goal and isn’t sure exactly what they need — a common situation for new hires or people starting on a new project. High-intent discovery happens when someone knows precisely what they’re looking for, such as a dataset with a specific field or a table used by a particular team.
The Lexikon homepage was rebuilt to support low-intent discovery with personalized, algorithmically driven recommendations. Users now see popular datasets across the company, recently used tables, datasets popular within their teams, and suggestions they haven’t used yet. The team experimented with NLP and topic modeling on dataset metadata for content-based recommendations, but found through feedback that simpler heuristics based on consumption stats worked better. Today, 20% of monthly active users reach BigQuery tables via homepage recommendations rather than search.
High-intent discovery required a different approach. Since a small share of top datasets accounted for the majority of consumption, search ranking was reweighted toward popularity. Users reported that results felt more relevant, and they gained confidence knowing a dataset was widely used within Spotify. Beyond ranking, Lexikon introduced new entity types — schema fields, BigQuery projects, people, and teams — to open up alternate discovery paths. For instance, searching for track_uri now leads to a schema field page listing the top tables containing that field. These new entity pages have become a critical pathway, drawing 44% of monthly active users.
Making people part of the search
The first version of Lexikon followed a codification strategy, assuming that well-documented datasets would eliminate the need for person-to-person knowledge exchange. In practice, data scientists still wanted to connect with colleagues who had expertise in a given area. Research into knowledge management theory reinforced the value of connecting people with the right datasets.
To make that connection easier, Lexikon added expert search. Users can search for people working on a given topic, with results ranked by a combination of production and consumption activity: owning datasets or dashboards, authoring research, running A/B tests, and querying or viewing related assets. Insights production activity is weighted more heavily than consumption. This is especially useful for newer employees who haven’t yet built informal networks across the insights community.
Data scientists also continued discussing datasets in Slack, so Lexikon shipped a Slack bot that summarizes shared dataset links. When a user posts a Lexikon dataset link, the bot surfaces the name, owner, description, usage stats, lifecycle information, access tier, most-used schema fields, and links to view details, request access, or open the table in BigQuery. The bot boosted weekly Lexikon link sharing in Slack by 25%, while also serving as an adoption channel for the tool itself.
Closing the final gap: from discovery to first query
Finding the right dataset is only half the battle. The harder step, as user research revealed, is figuring out how to actually start using it. Data scientists told Spotify’s team that uncertainty about a dataset’s schema fields, practical usage patterns, and join relationships often delayed their first query. Lexikon now ships three features aimed at that last mile.
Schema-field consumption statistics
Datasets can carry dozens or even hundreds of fields. Once a dataset looks promising, a data scientist faces the daunting task of deciding which fields matter. Lexikon augments the basic schema metadata with column-level usage stats: the number of queries referencing each field and the number of unique people who have queried it. Users can sort available fields by these consumption counts, making it easier to surface the fields most likely relevant to their task.
Queries
Curiosity about how a dataset works in practice is common. Lexikon’s initial answer was producer-submitted example queries, but that approach had two flaws: getting producers to contribute examples for every dataset was an uphill battle, and the examples tended to go stale as the underlying data landscape shifted (for instance, a join to a deprecated table would quietly break the sample).
The team replaced curated examples with a searchable log of all recent queries run against the dataset. This gives users a live, varied view of real usage patterns and lets them filter for specific cases, such as queries that reference a particular field. Since this feature shipped, 25% of users visiting a dataset page have used it.
Tables commonly joined
One dataset rarely answers an entire question. To help users find the other pieces of the puzzle, Lexikon’s BigQuery table pages now show which tables are most frequently joined with the dataset in question. Even though it’s not the most-used feature on the page, 15% of visitors consistently interact with it.
Impact on adoption and use
The improvements—understanding user intent, enabling knowledge transfer through people, and supporting the first steps with a discovered dataset—have shifted data discovery from a known bottleneck to a background strength. Adoption of Lexikon among data scientists climbed from 75% to 95%, landing it in the top five daily tools for that group. More data scientists report using Lexikon than the BigQuery UI, Python, or Tableau. Organic growth has also extended the tool’s reach beyond insights specialists: monthly active users grew from roughly 550 to roughly 870, including engineers and data-savvy product managers. Engagement deepened as well, with average sessions per monthly active user rising from about three to about nine since launch. In surveys, data discovery no longer appears as a primary pain point in insights production.
