A unified front end for SQL analysis at Meta
SQL analytics at Meta spans recurring pipelines and ad-hoc exploration across multiple query engines. That breadth created two persistent problems. First, static analysis of SQL—checking queries before execution for performance issues, data lineage, or simple mistakes—was impractical because each engine embeds its own analysis logic, tied to its own SQL dialect. Second, the type system used to describe warehouse columns was too limited, allowing errors like joining two user_id columns that actually come from different ID systems.
To address both, Meta built UPM (Unified Programming Model), an internal library that parses SQL into a semantic tree—a hierarchical data structure other tools can inspect or modify. Given a query, UPM returns the tree; downstream consumers use it for linting, rewriting, or even as a pluggable SQL front end for execution. UPM can also render a semantic tree back into a target SQL dialect and hand it to a query engine.
The semantic tree approach supports three distinct operations:
- Static analysis—inspect the tree to emit diagnostics or warnings, as a SQL linter does.
- Query rewriting—modify the tree to transform the query itself.
- Query execution—use the tree directly to generate and run a query plan, or render it back into SQL for the engine.
One language, many engines
UPM gives Meta a single language front end for SQL users: a superset of the Presto SQL dialect that works whether the target engine is Presto, Spark, or XStream, Meta's in-house stream processing service. Rather than reimplementing analysis for each engine's dialect, infrastructure teams use UPM semantic trees as a standard interchange format. This mirrors the role Velox plays as a pluggable execution engine—UPM is the pluggable language front end, saving teams from maintaining their own SQL parsers and analyzers.
Catching type errors before execution
Beyond dialect unification, UPM strengthens type checking. Every warehouse column carries a physical type like integer or string. Columns can also be annotated with optional user-defined types—Email, TimestampMilliseconds, UserID—that carry semantic meaning without affecting on-disk encoding. UPM uses these annotations to catch errors that physical types alone cannot.
Consider a UNION of two tables logging login events. One table records timestamps in milliseconds, the other in nanoseconds. Both columns share the name timestamp, so the query looks correct, but the schema annotations reveal the mismatch. UPM's typechecker flags the error in the author's editor before the query ever reaches an engine—a mistake that would otherwise run silently and surface much later.
Column-level lineage at warehouse scale
Meta's data lineage team built its query analysis tool on UPM semantic trees. The tool scans all recurring SQL queries to construct a column-level lineage graph across the warehouse. For each query, it deduces edges showing how data flows between input and output columns.
INSERT INTO user_logins_daily_agg
SELECT
DATE(login_timestamp) AS day,
COUNT(DISTINCT user_id) AS n_users
FROM user_login_events
GROUP BY 1
The lineage analysis derives edges like these:
[{
from: “user_login_events.login_timestamp”,
to: “user_login_daily_agg.day”,
transform: “DATE”
},
{
from: “user_login_events.user_id”,
to: “user_logins_daily_agg.n_user”,
transform: “COUNT_DISTINCT”
}]
Combined across every daily query, the result is a global view of data flow, which answers questions about data quality ("Where did this bad value come from?"), incident impact ("Which downstream assets depend on this corrupted table?"), and refactoring ("Can this table be safely deleted?").
Gradual rollout ahead
Most tables in Meta's Hive warehouse already carry user-defined types, and stricter type-checking is being enabled gradually to let existing pipelines migrate without breakage. UPM is now integrated into the primary SQL authoring surfaces at Meta. The long-term goal is for UPM to become Meta's unified SQL front end—integrated into all query engines, exposing a single dialect—while raising the abstraction level for query authors with conveniences like trailing commas in SELECT clauses and SELECT * EXCEPT <some_columns> syntax.
