Data Lineage at Slack

Why lineage became critical at Slack

Slack’s data platform has grown alongside the company itself, and with that growth came a practical problem: no one could easily trace how tables were produced and consumed. In the early days, a handful of pipelines meant relationships were simple enough to keep in someone’s head. As datasets multiplied, that stopped being true.

Knowing the full graph of data dependencies makes routine maintenance like backfills far less painful. For a table with no downstream consumers, a backfill is just rerunning the affected dates. For a table that feeds others, the scope can balloon — every downstream table may need attention, depending on which columns were touched. This was a “nice to have” problem until GDPR made it existential. GDPR’s Right to Know and Right to be Forgotten require companies to locate and act on all data about a given person. Without reliable lineage, identifying every relevant table across thousands of ad hoc datasets becomes a compliance nightmare.

Collecting lineage from pipelines and dashboards

Slack built a central Lineage Service to handle ingestion, parsing, and persistence. Raw lineage data lands in RDS MySQL tables; a separate process flattens that data and stores it in Hive. The service also exposes endpoints so various tools can submit metadata describing what jobs read and write.

Most of Slack’s data pipelines run on Airflow. The team integrated lineage ingestion directly into their DAGs using Airflow callbacks that fire after a successful task run. That avoids spam from failed or retried tasks. Because nearly every pipeline is scheduled through Airflow, this one integration captures lineage for the majority of the warehouse automatically.

Dashboards are a notable exception. They run Presto queries outside of Airflow, so Slack relies on audit tables that track dashboard usage. If a dashboard has been active in the last 24 hours, it gets treated as “active” and its SQL payload is submitted to the lineage service. Since the payload includes SQL, the parsing logic is nearly identical to the Airflow path.

Typical requests to the lineage service:

Parsing SQL into relationships

The core of the system is SQL parsing, done with ANTLR. The metadata sent with each request includes the DAG and task names, and since Slack’s DAGs make heavy use of sensors, that metadata alone is enough to assemble a graph of dependencies between DAGs. But the real value comes from extracting lineage from the SQL itself. A large suite of unit tests guards against regressions as the parser evolves.

Because SQL drives the vast majority of pipelines, this approach captures lineage for almost every table without manual documentation. That matters: hand-written lineage descriptions go stale in large repos and eventually do more harm than good. Deriving lineage from the actual SQL avoids that failure mode.

Querying lineage at scale

The raw RDS tables only capture direct dependencies. Answering a question like “what’s downstream of messages?” requires walking multiple hops manually. A query for direct consumers of messages returns messages_derived, but finding the full impact set means repeating the query for each new table until the graph is exhausted. That approach does not scale.

To solve this, Slack produces a flattened table in Hive on a daily cadence. The flattening job is a standard Spark process, with the interesting part built on Pregel: the job visits each node, gathers its lineage, recurses through every connection until the graph is fully traversed, then condenses the results into a plain dataset with source, target, and a layer field indicating steps from the source. The layer concept enables single queries for “everything two hops downstream of this table” or any other depth.

With the flattened table, the previous example becomes a single query.

Notifying downstream consumers

The Data Portal exposes this lineage data through a “notify” button. Dataset owners see who their downstream consumers are — dashboard creators, other dataset owners — and can send them a Slack message with a call-to-action and custom fields. This is how Slack coordinates data retention programs, planned deprecations, and other changes that ripple through the warehouse. The notification configuration and the resulting Slack message show how the tooling connects the lineage graph to real workflows.

What’s next

Three areas are on the roadmap. First, non-SQL Spark jobs that use the Datasets or Dataframes API have no SQL payload to parse, so lineage has to be added manually — a process that doesn’t scale and erodes trust. A new endpoint and parser would bring those jobs into the automated graph. Second, moving lineage data into a graph database would enable visual exploration of dependencies, making gaps more obvious and opening the data to less technical users. Third, better automation around schema changes: automatically detecting a changed schema and proactively notifying downstream consumers would tighten coordination between producers and everyone who depends on their tables.