How Meta discovers data flows via lineage at scale

Why Meta needed automated data lineage

Meta’s Privacy Aware Infrastructure (PAI) initiative relies on data lineage as a core discovery mechanism. It gives engineers a scalable way to trace where data originates and where it flows, which is a prerequisite for enforcing privacy controls such as purpose limitation. Without accurate lineage, it is impractical to identify every place user data is processed across Meta’s large, constantly changing codebase of billions of lines.

PAI builds on a foundation of three prerequisite steps that together form a unified asset catalog:

• Inventorying collects all code and data assets—endpoints, tables, AI models—used across Meta.
• Schematization describes the structure of those assets, such as indicating that a table has a religion field.
• Annotation labels asset content, for example identifying that a certain column contains religion data.

These steps took Meta years to complete across millions of assets, but they make lineage possible. The example used throughout this discussion is the religion field in the Facebook Dating app.

How data lineage fits into the PAI workflow

Lineage answers a simple question: “Where does my data come from and where does it go?” It tracks data as it moves from a source asset, such as a database table, to a sink asset in another system. This flow graph is the basis for implementing Policy Zones, Meta’s information flow control technology, and provides three main benefits:

• Scalable data flow discovery: Lineage graphs give an end-to-end view of how relevant data moves from collection points through all processing steps, making it possible to visualize flows for large-scale systems.
• Efficient rollout of privacy controls: With lineage, developers can locate the best places in the codebase to insert Policy Zones. The Policy Zone Manager (PZM) tool, built on lineage data, lets engineers identify multiple downstream assets at once, accelerating privacy control deployment.
• Continuous compliance verification: After privacy controls are in place, lineage remains useful for monitoring and validating that data flows stay within expected boundaries, complementing Policy Zone enforcement.

Collecting signals for lineage

Traditional approaches—manual data flow diagrams and spreadsheets—do not scale to Meta’s environment. Instead, lineage is assembled from two signal types: static code analysis performed across various programming languages, and runtime instrumentation that observes data as it flows through systems. These signals are combined into lineage graphs that developers can query through an intuitive user interface, eliminating the need to hand-trace data paths and reducing the engineering effort required for privacy work.

Lessons from scaling lineage

Expanding PAI across Meta surfaced several insights about building lineage infrastructure:

• Prioritize data lineage early in the privacy engineering lifecycle—it underpins later controls and is harder to retrofit.
• Invest in developer tooling with a usable interface to make lineage actionable, not just available.
• Design a cohesive set of libraries and schemas so that different signal collection techniques produce consistent, combinable lineage data.

Two-Stage Lineage for Sensitive Data

Meta's automated lineage tracking for religion data operates in two stages. First, engineers collect data flow signals from processing activities across the company's systems — not just for religion but for everything — building an end-to-end lineage graph. Second, they identify the specific subgraph within that graph that pertains to religion. The approach spans function-based systems (web services and backends using Hack, C++, Python) and batch-processing systems (data warehouses and AI pipelines using SQL).

Function-Based Systems: Capturing Web Flows

A Facebook Dating user populating their religious views sets off a chain of data movement. The religion value travels from mobile device to a web endpoint, is logged to a table, and stored in a database. The code paths touch logging frameworks and database drivers interchangeably across Meta's repositories.

To pin down flows in this function-call stack, Meta combined static analysis with runtime instrumentation. Static analysis simulates execution and maps flows, though it lacks runtime data, which can mean false positives from code that never actually runs. To close that gap, Meta's Privacy Probes collect signals at runtime from instrumented core frameworks and libraries — tracking data from origins (sources) to outputs (sinks) like loggers and databases.

Privacy Probes operates at request runtime in three passes:

1. Captures payloads: Samples source and sink payloads into memory with metadata — timestamps, asset IDs, stack traces — as evidence of a data flow.
2. Compares payloads: Checks source against sink payloads within a request to spot matching data.
3. Categorizes results: Partitions findings into a match-set of high-confidence flows where sink data exactly matches or is contained by source data, plus a full-set of all source-sink pairs in the request, which is a noisier superset that may still capture transformed flows. The full-set goes to human reviewers.

Given varied religion inputs at an endpoint, only some logged values will be exact or contained matches. A transformed value — for example, a religion string turned into a count before being logged — shows up in the full-set rather than the match-set.

These signals are assembled into a lineage graph that connects source and sink assets rather than individual data values.

SQL Engines: Tracing Warehouse and Batch Pipelines

Data flows differ in the warehouse, where Presto and Spark compute engines log SQL queries for data processing activities. Rather than instrumenting execution payloads directly, Meta performs static analysis on those logged queries and job configs. A SQL analyzer extracts flows between input and output tables at the table level as well as column granularity, such as user_id to target_user_id or religion to target_religion.

Sometimes SQL logs capture only a read or a write, not both. Meta connects those fragments using runtime metadata — execution environment identifiers and job or trace IDs — to fill the gaps, yielding a cleaner end-to-end warehouse graph.

AI Systems: Tracking Through Models

For AI assets, lineage comes from parsing training configs retrieved from the model training service. A config might bind an input dataset like asset://hive.table/dating_training_tbl and a feature like asset://ai.feature/DATING_USER_RELIGION_SCORE to a model such as asset://ai.model/dating_ranking_model.

More instrumentation catches relationships at load time across data-ingestion layers such as DPP, PyTorch libraries, the FBLearner Flow workflow engine, and inference systems. Backend service lineage reuses the function-based probing approach.

Pruning the Graph to What Matters

Once the full lineage graph exists, the challenge is isolating flows relevant to a given privacy requirement. Meta's iterative analysis tool combines the graph with Policy Zones privacy controls so developers can systematically narrow down the relevant subgraph:

1. Discover candidates: Start from source assets and follow flows downstream until hitting low-confidence nodes, marked yellow.
2. Include or exclude: Developers or heuristics mark red nodes to exclude (those without religion data) and green nodes to include. Culling red nodes early prunes their entire downstream branches at once. Policy Zones are used to make sure all potentially defining flows get captured.
3. Repeat: Begin again with green nodes as new sources until no further green confirms appear.

With this final subgraph in place, developers can proceed to apply privacy controls to safeguard sensitive religion assets.

Lessons from Building Lineage at Scale

• Do lineage early: Investing in deep data-flow understanding accelerated Policy Zones adoption and opened uses beyond privacy, such as security and integrity.
• Build for consumers, not just the data: Raw lineage signals were overwhelming for system owners. Iterative discovery tools cut human effort by orders of magnitude.
• Integrate via libraries: Asking each system to manually collect lineage signals stalled progress. Embedding lineage logic into PAI libraries in Hack, C++, and Python allowed much smoother adoption.
• Measure what you cover: Tracking lineage coverage continuously keeps the system current with new data and code, anchoring privacy outcomes to maintainable improvements.

What's next for lineage at Meta

Data lineage remains a core pillar of Meta's PAI initiative, giving engineers a clear picture of how information moves through internal systems. The foundation is in place, but the work continues across three fronts:

• Expanding coverage: extending lineage tracking to more systems and data flows so the map of data movement stays complete.
• Improving consumption experience: making lineage data easier for developers and stakeholders to query and act on.
• Exploring new frontiers: identifying additional applications for lineage, from compliance to engineering efficiency, and sharing those findings with the wider industry.

The goal is straightforward: by deepening lineage capabilities, Meta hopes to strengthen privacy awareness internally and contribute to a more transparent, accountable data ecosystem across the industry.

Acknowledgements

The authors thank the many current and former Meta employees who contributed to the development of data lineage technologies over the years. Special thanks go to (in alphabetical order) Amit Jain, Aygun Aydin, Ben Zhang, Brian Romanko, Brian Spanton, Daniel Ramagem, David Molnar, Dzmitry Charnahalau, Gayathri Aiyer, George Stasa, Guoqiang Jerry Chen, Graham Bleaney, Haiyang Han, Howard Cheng, Ian Carmichael, Ibrahim Mohamed, Jerry Pan, Jiang Wu, Jonathan Bergeron, Joanna Jiang, Jun Fang, Kiran Badam, Komal Mangtani, Kyle Huang, Maharshi Jha, Manuel Fahndrich, Marc Celani, Lei Zhang, Mark Vismonte, Perry Stoll, Pritesh Shah, Qi Zhou, Rajesh Nishtala, Rituraj Kirti, Seth Silverman, Shelton Jiang, Sushaant Mujoo, Vlad Fedorov, Yi Huang, Xinbo Gao, and Zhaohui Zhang. Appreciation also goes to the post reviewers (in alphabetical order): Aleksandar Ilic, Avtar Brar, Benjamin Renard, Bogdan Shubravyi, Brianna O'Steen, Chris Wiltz, Daniel Chamberlain, Hannes Roth, Imogen Barnes, Jason Hendrickson, Koosh Orandi, Rituraj Kirti, and Xenia Habekoss. The authors especially thank Jonathan Bergeron for overseeing the effort and providing guidance, Supriya Anand for leading the editorial direction, and Katherine Bates for coordinating the support needed to publish this post.