A Format Change at Exabyte Scale
Meta’s data platform has grown well beyond what its original logging formats were designed to handle. The legacy wire serialization formats — Hive Text Delimited and JSON — began showing concrete reliability and efficiency problems under the scale demands placed on the platform. These issues prompted a rethinking of how data is logged, from first principles, and led to the development of the Tulip serialization format as the replacement.
The migration to Tulip was complicated by two factors beyond the technical work itself: years of accumulated technical debt that created deep dependencies between systems, and the nontechnical challenge of guiding thousands of engineers through a change to their logging workflows. To manage these, the team broke the migration into two phases: a planning and experimentation phase to build and validate technical solutions, and a scaling phase focused on automation, self-service tooling, and reducing friction for end users.
Why Move Away from Legacy Formats?
The logger is the heart of Meta's data platform, feeding analytical and operational data to systems like Scuba, Hive, and stream processing pipelines via Scribe. Every product and data platform team relies on it. The data format used for logging was Hive Text Delimited or JSON for legacy reasons; the limitations of these formats drove the design of the Tulip serialization format as a replacement for those destination-specific serializations.
Two-Stage Migration Strategy
The migration journey was split into two phases, each with a distinct focus:
- Planning, preparation, and experimentation: Building automation and validation tools to enable a safe transition. Data consumers had to be migrated before producers, with a small number of critical teams undergoing white glove migrations first to inform the strategy.
- Scaling: Based on learnings from the first phase, the team built self-serve tooling and features designed for efficient interactions with a large number of engineering teams.
Engineering for a Smooth Wire Format Switch
The engineering solutions fell into two main categories: wire format design and testing/rollout tooling. An early design requirement was avoiding the need for producers and consumers to switch serialization formats atomically.
Supporting Mixed Formats
When a logging schema was flipped to use Tulip, it was impossible to switch all producers to the new format simultaneously. To allow a gradual, rate-limited rollout, the team needed to support mixed format payloads on a single Scribe stream. The key constraint was that existing payloads could not be changed, so every Tulip-serialized payload was prefixed with a 2-byte sequence (0x80 0x00) that is invalid UTF-8. This made each payload self-identifying and allowed the different formats to coexist within the same stream.
Data Consumers
Some applications had been written to rely on the Hive Text or JSON structure of the data, which was a consequence of consumers breaking through the serialization abstraction. Two approaches addressed this.
Reader library: The counterpart to the logger is a Reader library that converts a serialized payload into a structured object. Available in code-generated and generic forms, a reader object consumes data in any of the three formats — Tulip, Hive Text, or JSON — and produces the structured object. Consumers could be switched over to use readers ahead of the migration, and application code was updated to consume the structured object instead of a raw serialized line. This made the wire format invisible to consumers.
Format conversion: For use cases where updating application code was too costly, the consuming system was equipped with a converter that accepted Tulip-serialized payloads and converted them back to Hive Text or JSON. This was inefficient in CPU usage, but it let the migration proceed even for a long tail of legacy use cases.
Validation and Rollout Tooling
Once the wire format was prepared, the focus shifted to finding issues proactively and stopping the bleeding quickly if something went wrong in production.
Shadow Loggers
To perform end-to-end testing and verification before a full rollout, the team created "shadow loggers." These mimicked the original logging schemas but wrote to dedicated tables monitored by the logger team. A shadow logging schema appended two additional columns to the user-specified fields:
- Serialization format: Which format was used to write the row (Hive Text or Tulip).
- Row ID: A unique identifier for the row that connects the same logical record serialized by each format.
Shadow loggers captured a small fraction of the rows on every log to the main schema. A Spark job then compared rows with the same ID to verify the contents were identical across the differing serialization formats — a critical end-to-end confirmation that ran before any large-scale rollout.
Debugging with Loggertail
To test and validate data after migration, the team built a CLI tool called loggertail. The tool queries the serialization schema for a named logging schema and uses a generic deserializer to decode the message. It prints the results as a human-readable JSON object, which allows quick inspection of a logging schema's Scribe queue post-migration.
Rate Limiting the Rollout
Even with careful validation, unforeseen problems can surface in a live system. A rate limiter was built so teams could roll out Tulip serialization to a schema incrementally and, if a problem emerged, swiftly shut off or throttle the flow of new-format payloads.
Scaling to 30,000+ Schemas
The next phase moved to self-serve automation to handle a broad migration with minimal manual effort. This required categorizing schemas by the downstream consumers of their corresponding Scribe streams. Only schemas whose entire downstream consumer list was already upgraded to read the Tulip format were eligible for migration.
Building on that categorization, a tool was developed that automatically targeted the unmigrated logging schemas. An engineer needed only to run the script. The automation tooling also detected potential data loss scenarios, which gave the team a regular, low-touch migration cadence. Eventually, these scripts were run automatically via a cron-like scheduling system.
Human Factor of the Migration
Expert guidance and support chains played a major role in accelerating the work. Teams were given lead time via tasks, and we produced a live migration guide, which is different from a static how-to. The guide constantly evolved as various teams across Meta went through the process — their shared experiences shaped it as newer, better ways of migrating emerged. A short demo video was also made to walk through the mechanics of migrating a logger end to end.
To unblock individual teams, we ran a support group and office hours where users could post blockers, get solutions, and share wisdom about migrating specific logging scenarios. This community-driven support became the go-to source for solving the long tail of migration edge cases faced across the company.
Impact of the Migration
Across the platform, the number of logging schemas remained roughly the same while the bytes logged fell noticeably when measured across the conversion effort.
The savings are attributable to the change in serialization format, with the migration journey reflected in chart milestones. The numbers are extrapolated to overall traffic based on measurements from the five largest (by volume) logging schemas.
Beyond the metrics, the switch to Tulip has changed how data flows through Meta's platform — abstracting the wire format for consumers, moving operational pipelines toward self-serve automation, and eliminating legacy serialization bottlenecks for systems serving the entire company. The project not only solves the immediate format-specific issues faced today but also makes future platform-scale migrations and systems healthier because they are no longer working from the constraint of a format that has outlived their scaling limits, even when tackling the task has been a hard road working through dependencies at every level.
Long-term gain, short-term pain
Undertaking a cross-platform serialization format change is a significant short-term investment with long-term payoff. The key is building solutions that account for both the technical and operational realities of migrating infrastructure of this scope.
For Meta, the success of this project depended on tight collaboration between the data platform and logger teams, plus buy-in from the engineers who use the systems daily. A migration of this magnitude cannot succeed in isolation — it requires cross-functional alignment and a willingness to evolve infrastructure over time.
