Why Facebook built Nemo
At Facebook’s scale, finding the right data artifact quickly is a constant operational challenge. Engineers, product managers, and production staff all need to locate relevant tables, dashboards, or datasets to make informed decisions—but the path to that information is often unclear. Tables may have obscure names, multiple teams may maintain overlapping datasets, or a dashboard may have been silently superseded months ago.
Other large companies like Airbnb, Lyft, Netflix, and Uber have built their own discovery tools for this problem. Facebook’s answer is Nemo, an internal data discovery engine designed to give engineers fast access to accurate results without routing every request through a human expert.
Facebook maintains more than a dozen artifact types, including Hive tables, Scuba tables, dashboards, AI datasets, and Cubrick. Before Nemo, internal surveys showed that finding the right data was a major pain point. Since its rollout, search success rates have improved by over 50 percent, even as the total number of artifacts has more than tripled and query volume has more than doubled.
What makes discovery hard
The problem is not just about scale—it’s about context. A data engineer investigating latency impact, a product manager tracking regional usage trends, or a production engineer grouping task crashes by team all face the same challenge: the data exists, but knowing where to look is often knowledge held by a few senior employees. Even knowing how to phrase a query is nontrivial.
Facebook’s original solution relied on an Elasticsearch cluster indexing metadata such as creator, size, column structure, and descriptions. Searches were plain text. As datasets grew, search quality degraded, and the gap between data creators and consumers widened.
Nemo replaces this foundation with several key upgrades:
- Scalable storage: Nemo uses Unicorn, the same search infrastructure that powers Facebook’s social graph search.
- Advanced querying: Users can restrict searches by criteria like monthly dashboard views or data freshness, and filter out artifacts with privacy restrictions.
- Natural language parsing: Queries like "How many weekly active users are there on Instagram?" return pointers to the relevant table, subject to existing permissions.
- Personalization: Ranking accounts for an engineer’s role and team.
- Post-ranking model: A gradient-boosted decision tree trained via FBLearner incorporates a broad set of metadata signals.
- Trust signals: Quality, recency, usage, and lineage data influence ranking, steering users toward reliable artifacts.
Architecture: indexing and serving
Nemo has two main components: indexing and serving, with a front end layered on top of serving. Indexing splits into bulk indexing (daily) and instant indexing (immediate). When a Hive table is created, an instant update makes it findable by name or creator within seconds. Heavier signals, such as the number of engineers who accessed a table over the past month, are collected in the bulk process and may lag by a day or two.
Team members who create new artifact types can make them searchable themselves by calling Nemo APIs. Indexing is also aware of the broader data ecosystem: if a pipeline duplicates a column into a downstream table, the original column description and upstream table name are stored with the downstream artifact. Presto queries are logged—an engineer running a query against a table increases its score both globally and for that specific engineer.
Serving pipeline
On the serving side, a spaCy-based NLP library handles text parsing. Unicorn manages retrieval and initial ranking. Post-processing applies more sophisticated signals, including kNN-based scoring and the FBLearner-trained model, along with social signals like an artifact’s user list. Textless queries—just a set of type and quality filters—are handled specially, with final scores emphasizing personal and team-level usage.
The front end displays results and supplies a query-building system that translates user-specified restrictions into Unicorn queries. It also flags duplicated or low-quality artifacts to help users avoid picking the wrong source.
Signals and training
Nemo’s ranking signals range from simple textual overlap between artifact name and query, to content-aware measures like widget counts in a dashboard, to personalized signals such as how many people with your role accessed a table recently. A query-independent trust score assesses reliability based on usage and freshness heuristics.
For model evaluation, Nemo counts more than clicks: if a user views an artifact and then accesses it later that same day, that is considered a positive signal. This richer interaction data improves training quality, though it comes with complications.
Lessons learned
Internal users are orders of magnitude fewer than external product users, so A/B tests on Nemo iterations took days or weeks each, making parallel testing impractical. Labeling results was also difficult—judging correctness often requires domain-specific knowledge, and the volatile dataset means labels decay quickly. Building "golden sets" for testing and training was a periodic but essential investment.
Not all data can stay current simultaneously, so Nemo’s ranking plays a role in code and data health by reducing the chance engineers adopt outdated artifacts. Over time, Nemo has shifted from raw search terminology toward user intent, mirroring the trend of direct answers in modern search. It is now used by tens of thousands of engineers as a regular part of their workflow—a scalable foundation for data discovery as Facebook continues to grow.
