A shared in-memory format for Velox and Arrow

Meta’s Data Infrastructure teams have been decomposing monolithic data management systems into reusable components, and open standards are central to that effort. Apache Arrow 15.0.0 now includes three new columnar layouts—StringView, ListView, and Run-End-Encoding—developed in partnership with Voltron Data and the Arrow community. These layouts were inspired by Velox, Meta’s open source C++ execution engine, and they aim to close the interoperability gap between the two formats.

The motivation is practical. Meta’s engines handle offline ETL, interactive dashboards, ad hoc exploration, stream processing, and increasingly AI/ML workloads. Rather than re-implementing common features in each engine, Velox provides a unified execution library that is currently integrated into more than ten Meta data systems, with 3-10x efficiency gains observed in Apache Spark and Presto integrations. Velox now counts over 200 contributors from more than 20 companies.

Where Arrow and Velox diverged

Arrow is an open standard for columnar in-memory data; Velox adds execution and resource management primitives on top. Both represent a column as a contiguous value buffer plus an optional validity bitmask. They already agreed on fixed-size scalar types and dictionary encoding, but diverged on strings, variable-sized containers, and encodings for constant or repeated values.

Velox initially extended Arrow into a proprietary layout called Velox Vectors to accelerate common processing patterns. That delivered speed but created fragmentation. The new Arrow layouts restore compatibility while preserving Velox’s performance characteristics.

StringView for efficient string processing

Arrow’s original string layout uses a data buffer plus an offsets buffer, where the length of string i is offsets[i+1] - offsets[i]—essentially an array of characters. The new StringView layout instead stores a 16-byte view object per string. The first four bytes hold the length. Strings up to 12 characters are stored inline; longer strings store a 4-byte prefix, a buffer ID, and an offset into a data buffer.

This design yields four benefits:

  • Short strings are read directly from the views buffer without dereferencing the data buffer, improving memory locality.
  • String comparisons can fail fast on the 4-byte prefix, avoiding data-buffer access for selective filters and sorts.
  • Operations like trim() and substr() can be zero-copy by updating only the view object.
  • Fixed-size views allow out-of-order writes.

Other engines, including Umbra and DuckDB, had already adopted a similar representation, so StringView in Arrow 15 enables efficient batch transfer across these systems as well.

ListView for variable-sized containers

Arrays and maps in Arrow originally used a flattened element buffer plus an offsets buffer; row i’s length is offsets[i+1] - offsets[i]. But Velox needed out-of-order writes to efficiently execute vectorized conditionals like IF and SWITCH. That requires knowing the length of each row’s container ahead of time, so Velox stores both lengths and offsets buffers.

Arrow 15 adds ListView, which supports the same dual-buffer representation. Beyond out-of-order writes and efficient conditionals, ListView enables zero-copy slicing and rearrangement such as slice() and trim_array(), and it permits containers with overlapping element ranges.

Run-End-Encoding for repeated values

Velox also introduced constant encoding—for columns where every value is the same, such as literals or partition keys—and run-length encoding (RLE) to compactly store consecutive repeated elements. Arrow 15 incorporates a variation called Run-End-Encoding. Instead of storing each run’s length, REE stores the offset marking each run’s end, which improves random access. Constant-encoded columns can be represented as a single run covering the entire batch.

Toward composable systems

The new Arrow layouts enable zero-copy, in-memory communication between Velox components and Arrow-based components. Adoption is underway in libraries like PyArrow and Polars, and within Meta. This convergence should also ease integration between projects such as Apache Gluten, which uses Velox internally, and PySpark, which consumes Arrow.

By aligning on open standards for in-memory data, Meta and the broader community reduce duplicated effort and fragmented tooling. The goal is a modular stack of reusable, open source components that can be combined more freely—and a foundation that accelerates innovation in data management.

Format Harmonization: What Actually Changed

The alignment effort focused on a few concrete, high-leverage areas of the format spec. The goal was not to rewrite Arrow, but to close specific gaps that made Velox's behavior non-standard or ambiguous.

Null representation: The biggest change concerns how Velox handles nulls relative to the Arrow specification. Prior to this work, Velox followed a pattern where a null bit of 0 designated a null value, which matches Arrow. However, Velox also had a legacy mode where the bit meaning was inverted. That legacy path has been removed. Velox now consistently uses the Arrow convention: a set bit (value 1) means the value is non-null, and an unset bit (0) means the value is null.

Variable-length types: Prior to the alignment, Velox's handling of string and binary offsets occasionally differed from Arrow in edge cases. The representation now conforms to the Arrow spec's use of 32-bit or 64-bit signed offsets for VARCHAR, VARBINARY, and list types. The semantics for zero-length values and empty strings are now well-defined and consistent with Arrow.

Decimal types: Velox historical support for decimal types used a different internal scale. The alignment establishes a canonical mapping to Arrow's Decimal128 and Decimal256 types, with the fixed precision and scale parameters interpreted identically across both systems.

Struct and complex types: The handling of field names and nullability flags within STRUCT and MAP types was tightened. There were prior discrepancies in how nullability was propagated from the schema down to child fields. The alignment ensures the schema-level metadata is consistently applied to nested types.

Schema Validation and the "Strict" Mode

To enforce the new alignment and prevent regressions, the project introduces a more rigorous schema validation layer. This is not optional. Velox now validates incoming Arrow schemas against a strict set of rules that assert the type mapping and the structural invariants described above. If a schema does not conform to the Arrow spec—for example, if it uses an offset type that does not match the declared length of a buffer—the import fails fast with a descriptive error rather than producing silent corruption.

This strictness is rooted in the practicalities of data exchange. Arrow buffers are often shared without deep copies. If Velox reads a buffer assuming its own legacy semantics, it can misread every value in a column. By requiring conformance at the boundary, the system ensures that memory is only interpreted in one way, eliminating a whole class of integration bugs.

Performance Considerations

The shift to strict Arrow conformance was implemented with performance parity as a hard requirement. Moving from the legacy null convention to the spec-compliant one is a simplification in the code path. There is no runtime branch to check which convention is in use, and there is no pass to convert between representations.

The one notable performance consideration is the potential for zero-copy vs. copy-on-write semantics when constructing output. When Velox produces data in Arrow format, the framework now guarantees that the output buffer is fully compliant. This may, in certain suboptimal cases, require a copy when a legacy structure is detected at runtime. However, these paths are rare and are logged as warnings so users can identify code that still generates non-compliant buffers. The standard paths remain truly zero-copy, with no additional memory allocation or copying of code.

What This Enables

With the format fully aligned, the interoperability surface becomes much cleaner. Tools that emit Arrow can now feed Velox through a single, well-documented gateway with predictable behavior. Likewise, Velox can produce Arrow that is consumed by the wider ecosystem—including PyArrow, DuckDB, and other query engines—without requiring a compatibility shim in the middle.

The removal of legacy mode also simplifies production operations. Previously, debugging an integration bug required checking whether the producer and consumer agreed on the null bit convention. The alignment project removes that entire class of configuration and reduces the operational burden on teams that use both data formats side by side in the same cluster.

Acknowledgments: This format alignment was only possible due to a broad collaboration across different groups. A special thank you to Masha Basmanova, Orri Erling, Xiaoxuan Meng, Krishna Pai, Jimmy Lu, Kevin Wilfong, Laith Sakka, Wei He, Bikramjeet Vig, and Sridhar Anumandla from the Velox team at Meta; Felipe Carvalho, Ben Kietzman, Jacob Wujciak-Jens, Srikanth Nadukudy, Wes McKinney, and Keith Kraus from Voltron Data; and the entire Apache Arrow community for the insightful discussions, feedback, and receptivity to new ideas.