Common components, fewer silos

Meta’s data infrastructure has grown into a collection of specialized engines — batch ETL, interactive analytics, stream processing, and ML feature pipelines all run on their own systems. That breadth came at a price: little code sharing between engines, duplicated engineering effort, and inconsistent APIs and semantics that forced internal users to learn each system’s quirks. As the number of systems grew, so did operational cost, and adapting to new product requirements slowed down.

The response was a deliberate shift in engineering strategy. Instead of building each data management system as a monolith, Meta began identifying shared components, extracting them into reusable libraries, and standardizing the interfaces between them. The goal was threefold: reduce duplicated work, present users with more consistent semantics across engines, and accelerate the pace of innovation. This approach has since been articulated in a research paper co-authored with other organizations, which outlines the vision for a composable data management system.

A key observation from that work is that seemingly distinct data systems are built from a common set of logical layers:

  • A language frontend that parses user input such as SQL or dataframe code into an internal format;
  • An intermediate representation (IR), typically a logical or physical query plan;
  • A query optimizer that transforms the IR into a more efficient form for execution;
  • An execution engine library that runs query fragments locally (the eval engine); and
  • An execution runtime that supplies the distributed environment for running those fragments.

The data structures and algorithms behind these layers are largely consistent across systems. There is nothing fundamentally different between a SQL frontend for an operational database and one for a warehouse, or between the expression evaluation engines of a columnar DBMS and a stream processor. Domain-specific needs do exist — streaming operators, tensor manipulation — but these should be served by extensibility APIs layered over shared, common functionality. The cultural shift is to focus on the similarities, which are the norm, rather than the differences, which are the exception.

Extracting the execution layer

Refactoring mature, battle-tested systems is rarely practical. However, the execution engine is where most compute resources are spent, and Meta repeatedly found itself re-implementing optimizations that already existed in another engine, or porting features across systems. The decision was to build a new execution-layer component from scratch, designed as a composable and reusable library with all the optimizations needed across engines, rather than tweaking each system individually.

That component is Velox, created in late 2020 and open sourced in 2022. Velox is an engine- and dialect-agnostic execution engine, meaning it can be integrated into any data system and extended to match different SQL dialects. It began with collaborators from IBM/Ahana, Intel, and Voltron Data; today more than 200 individuals from over 20 companies contribute to its development.

At Meta, Velox is in various stages of integration with more than ten data systems. The integration with Presto, called Prestissimo, has demonstrated 3–10x efficiency improvements in production workloads. Intel’s Apache Gluten project, which uses Velox as the execution engine within Spark, shows a similar 3x gain on benchmarks. New systems — such as internal time-series databases and low-latency interactive engines — have been developed in record time by reusing Velox rather than building execution logic from scratch. For Meta, it has concentrated database execution expertise in a small, focused team whose work benefits many systems at once.

Velox achieves its performance through four core techniques:

  • Columnar and vectorized execution: large computations are broken into tight loops with predictable memory access patterns that modern CPUs handle efficiently.
  • Compressed execution: columnar encodings serve dual duty for data compression and processing efficiency. For example, dictionary encoding compacts data but also represents the output of operations like filters, joins, or unnests without materializing results.
  • Lazy materialization: many operations only wrap encodings around data, so actual decoding can be delayed or avoided altogether.
  • Adaptivity: Velox tracks runtime statistics across data batches — such as filter hit rates, join-key cardinality, and column access patterns — to reorder operations, organize joins, and improve prefetching on the fly.

Converging on open interfaces

Composable execution requires interoperable components. Engines must share file formats, serialization protocols, and table APIs, and they need a consistent way to express computation. In many cases this means transferring in-memory datasets across language boundaries, for example from C++ to Java or Python for efficient user-defined functions. Meta favors open standards for these interfaces.

Velox itself initially deviated from the Apache Arrow columnar format, creating a custom layout called Velox Vectors. The design accelerated common data-processing operations but fragmented the ecosystem and limited interop with Arrow-based systems. To close that gap, Meta partnered with Voltron Data and the Arrow community to align the two formats. After a year of work, Apache Arrow releases now include three extensions inspired by Velox Vectors: StringView, ListView, and Run-End-Encoding (REE). These additions enable zero-copy in-memory communication between Velox and Arrow components and broaden Arrow’s usefulness in modern execution engines.

Looking Ahead

Meta’s push toward composability has revealed two areas where the architecture needs to evolve further: file formats and hardware. On the file-format side, the rigidity of current encoding schemes is starting to constrain AI/ML training workloads. Those tables are often much wider than traditional analytics tables—sometimes by thousands of columns—and they can benefit from recursive, more flexible encodings as well as parallel decoding paths that keep up with data-hungry training pipelines. To address this, Meta created and open-sourced Nimble (formerly Alpha), a file format designed for large AI/ML datasets that also offers features useful for conventional analytic tables. Nimble is distributed as a portable library, with the expectation that it could eventually replace mainstream analytic file formats within Meta and elsewhere.

On the hardware side, AI/ML compute demands are pushing data center designs toward heterogeneity; specialized accelerators are becoming a bigger part of the infrastructure. The convergence between AI/ML and data management systems through hardware-accelerated data processing is a natural next step. Historically, fragmented software stacks hindered accelerator adoption in data management, but composable systems offer a more suitable architecture. In Meta's experience with Velox, the first 3-4x efficiency improvement came from software alone; the next 10x, the team believes, will need hardware acceleration. That work is still exploratory, with more open questions than answers, but two points are clear: composability is the enabling factor for widespread accelerator adoption, and open-source collaboration will be key to making it work.

The direction is set: the future of data management at Meta is composable — and the company hopes others will contribute to that vision.