Why Meta built a shared execution engine

Meta's data infrastructure spans dozens of specialized computation engines, each tuned for workloads ranging from batch and interactive SQL to transactions, stream processing, and data ingestion. The rise of AI and ML workloads has added further engines for feature engineering and preprocessing. Despite overlapping requirements, these systems evolved independently, leading to duplicated effort, inconsistent semantics, and difficulty adapting as hardware changes.

Velox is Meta's response: an open source, unified execution engine designed to be embedded in other data systems. It centralizes the data-intensive operations that execution engines have in common—expression evaluation, aggregation, sorting, joining, and similar work—so individual systems no longer need to reimplement them. Velox is under active development and is being integrated with more than a dozen data systems at Meta, including Presto, Spark, and PyTorch (via the TorchArrow preprocessing library).

Since the project was published on GitHub, it has drawn more than 150 contributors, including collaborators from Ahana, Intel, and Voltron Data, as well as several academic institutions.

Architecture and components

Velox sits at the data plane level: it receives an already-optimized query plan and executes it efficiently using local host resources. The engine applies runtime optimizations including filter and conjunct reordering, key normalization for hash-based operations, dynamic filter pushdown, and adaptive column prefetching, using statistics derived from incoming data batches. Complex data types are a first-class concern; the engine leans heavily on dictionary encoding for joins and filtering while retaining fast paths for primitive types.

Velox
Data management systems like Presto and Spark typically have their own execution engines and other components. Velox can function as a common execution engine across different data management systems. (Diagram by Philip Bell.)

Velox exposes the following building blocks to engine developers:

  • Type: a generic type system covering scalar, complex, and nested types such as structs, maps, arrays, lambdas, decimals, and tensors.
  • Vector: an Apache Arrow–compatible columnar memory layout with flat, dictionary, constant, sequence/RLE, and frame-of-reference encodings, plus lazy materialization and out-of-order result buffer population.
  • Expression Eval: a vectorized evaluation engine using common subexpression elimination, constant folding, efficient null propagation, encoding-aware evaluation, dictionary peeling, and memoization.
  • Functions: row-by-row and batch-by-batch APIs for building scalar and aggregate functions; a package compatible with the PrestoSQL dialect ships with the library.
  • Operators: implementations of TableScan, Project, Filter, Aggregation, Exchange/Merge, OrderBy, TopN, HashJoin, MergeJoin, Unnest, and other common SQL operators.
  • I/O: connector APIs for custom TableScan and TableWrite data sources, the DWIO interface for Parquet, ORC, and DWRF files, byte-level storage adapters for systems such as Tectonic, S3, and HDFS, and serializers supporting PrestoPage and Spark's UnsafeRow formats.
  • Resource management: primitives for CPU and memory management, spilling, and memory and SSD caching.

Real-world integrations

Three integrations illustrate how Velox unifies execution across otherwise disparate systems.

Presto — Prestissimo

Prestissimo, initially created by Meta in 2020 and developed with Ahana and other contributors, replaces Presto's Java workers with a C++ process built on Velox. The project implements Presto's HTTP REST interface—including worker-to-worker exchange serialization, coordinator orchestration, and status reporting—so a Java coordinator can offload plan fragments to native workers.

Benchmark results show near-order-of-magnitude speedups for CPU-bound TPC-H queries and 3-6x gains for shuffle-bound ones. In a production test replaying Meta's interactive analytical traffic, Velox delivered 6-7x average speedups, with some queries improving by more than an order of magnitude.

Velox
Prestissimo results on real analytic workloads. The histogram above shows relative speedup of Prestissimo over Presto Java. The y-axis indicates the number of queries (in thousands [K]). Zero on the x-axis means Presto Java is faster; 10 indicates that Prestissimo is at least 10 times faster than Presto Java.

Spark — Gluten

Gluten, created by Intel, lets Spark SQL delegate execution to C++ engines through a JNI API based on Apache Arrow data and Substrait query plans. Velox plugs into Spark simply by implementing that interface, decoupling the Spark JVM from the execution engine.

TorchArrow

TorchArrow is a PyTorch dataframe library that translates its representation into a Velox plan for execution. The integration converges ML data preprocessing under the same execution engine used for analytics, so users see consistent functions and UDF behavior across both domains. The project is available in beta on GitHub.

Industry implications

Velox's design suggests a shift away from monolithic, self-contained database systems toward reusable execution components. The unified stack also allows Meta and its partners to adapt the software proactively as hardware evolves. The project's documentation and full experimental details are published on the Velox website and in the paper "Velox: Meta's unified execution engine," presented at VLDB 2022.