A binary format for Meta’s data logging stack

Meta’s data platform is a collection of heterogeneous services — warehouse storage, real-time systems, and multiple ingestion pipelines — that exchange large volumes of data through service APIs. As AI and ML workloads grow, engineers are under steady pressure to make data logging more efficient while preserving reliability. Schematization is central to that effort: at Meta’s scale, thousands of engineers create, update, and delete logging schemas every month, with petabytes of data flowing through them daily over Scribe.

To address these pressures, Meta is sharing Tulip, a binary serialization protocol designed with safe schema evolution baked in. The protocol replaces legacy text formats used across the data platform and has delivered substantial gains in both size and CPU efficiency.

Why text formats failed at scale

The data analytics logging library sits in the web tier and in internal services, and is responsible for logging analytical and operational data through Scribe, Meta’s durable message queuing system. Downstream systems — including the Ingestion Service and real-time processors like Puma, Stylus, and XStream — read and consume this data. The library exists in two flavors:

  • Code-generated: statically typed setters for each field are generated for type safety, with post-processing and serialization code generated where possible for efficiency (e.g., Hack’s Thrift serializer uses a C++ accelerator).
  • Generic: a C++ library called Tulib performs (de)serialization of dynamically typed payloads. This mode allows (de)serialization without rebuilding or redeploying the application binary.

Historically, the logging library wrote data to multiple back-end systems, each dictating its own serialization mechanism. Warehouse ingestion used Hive Text Delimiters, while other systems relied on JSON serialization. Using either — or both — introduced several chronic problems:

  1. No standardization: each downstream system had its own format, raising development and maintenance costs.
  2. Reliability: Hive Text is positional. New columns can only be appended; adding or deleting columns in the middle shifts all subsequent ones and makes rows impossible to deserialize, since a row is not self-describing.
  3. Efficiency: both text-based protocols are inherently less efficient than binary (de)serialization.
  4. Correctness: Hive Text requires escaping and unescaping of delimiters and control characters. Handling legacy or buggy implementations that reject messages outright when they detect such characters adds further burden on library authors.
  5. No forward/backward compatibility: consumers needed to read payloads serialized by schema versions before and after their own, a guarantee Hive Text does not provide.
  6. No metadata: Hive Text does not easily allow attaching metadata to payloads, which is important for features like debug workflows that rely on hostnames or checksums traveling with the data.

The fundamental reliability problem — safe schema evolution — is what Tulip was designed to solve. The other issues could each have been addressed separately, but Tulip’s ability to resolve them all at once made the investment far more compelling.

How Tulip works

Tulip is a binary serialization protocol built on Thrift’s TCompactProtocol. Field numbering follows the same rules engineers already know from updating IDs in a Thrift struct. The key difference: when engineers author a logging schema, they specify field names and types, but field IDs are assigned by the data platform management module rather than by hand.

The workflow looks like this:

  • The engineer creates or updates a logging schema; the change is validated, then published to various data platform systems.
  • The logging schema is translated into a serialization schema and stored in the serialization schema repository.
  • A serialization config holds the ordered list of (field name, field type, field ID) for the logging schema, along with full field history.
  • Updates to a logging schema trigger a transactional operation on the serialization schema.

Schema evolution rules

Because IDs are managed centrally and never reused, the serialization schema maintains a complete field history. The evolution semantics are straightforward:

  1. Field addition: when a new field is added to the logging schema, a fresh ID is assigned in the serialization schema.
  2. Field type change: if a field’s type changes, the new field gets a new ID, while the original ID is retained. If the underlying data store does not allow type changes, the logging library blocks the modification.
  3. Field deletion: IDs are never removed from the serialization schema, preserving complete backward compatibility with payloads that were serialized earlier. Fields in the serialization schema are indelible.
  4. Field rename: there is no rename operation as such; it is treated as a deletion followed by an addition.

Measured impact

The efficiency gains come directly from moving to binary encoding. Compared with the previous Hive Text Delimited and JSON serialization formats, the Tulip format needs between 40 percent and 85 percent fewer bytes, and consumes 50 percent to 90 percent fewer CPU cycles to (de)serialize data. That is the kind of payoff that short-term complexity of changing serialization formats can bring — particularly at exabyte scale — since the benefits compound as the platform evolves.