Introducing OpenZL: An Open Source Format-Aware Compression Framework

A New Compression Middle Ground

OpenZL, now publicly released as an open source framework, targets a familiar problem: structured data compresses better when the compressor knows the structure. Format-specific compressors exploit that knowledge to beat general-purpose tools, but each bespoke scheme means another binary to build, ship, audit, and maintain. OpenZL tries to split the difference — format-aware compression on the encoding side, a single universal decoder on the other.

The framework applies a configurable sequence of reversible transforms that reveal hidden patterns in the data before entropy coding. Each file type can get a different transformation pipeline, yet every OpenZL file decompresses with the same universal binary. The decoder needs no out-of-band information; the transform recipe is embedded in each frame chunk.

From Zstandard to the Next Leap

When Zstandard launched, it paired strong entropy coding with CPU-friendly design to outperform its predecessor on both ratio and speed. Incremental work within that framework, however, offers diminishing returns. A recurring pattern emerged in the search for the next step: data is rarely a byte soup. It carries enums, ranges, repeated fields, columnar layouts, and other predictable shapes. A compressor that exploits those shapes can beat a general-purpose tool on both compression ratio and throughput.

OpenZL makes structure an explicit input rather than something the compressor must guess. Users provide a data shape via a preset or a thin description. An offline trainer then builds a compression config for similar data. At encode time the config resolves into a concrete graph embedded in the frame; the universal decoder executes that graph directly.

Format-Aware Compression in Practice

A useful illustration comes from the Silesia Compression Corpus, specifically the sao file, which holds an array of star records in a well-defined format. On an M1 CPU with clang-17, OpenZL achieves a higher compression ratio than generic lossless tools while preserving or improving speed — an important property for data center pipelines.

The gains come from a fairly simple processing graph:

The pipeline starts by splitting off the header, then transposing the array of structures into separate streams per field. Each stream holds homogeneous data of one type, so it becomes tractable to pick the right strategy:

• SRA0 (X-axis position) is mostly sorted due to how the table is generated. A delta transform shrinks the value range, making the stream cheaper to encode.
• SDEC0 (Y-axis position) is less sorted but bounded between known minimum and maximum values, so higher bytes are predictable; a transpose step exploits that.
• Fields like IS, MAG, XRPM, and XDPM have cardinality far below their count and no relation between consecutive entries. Tokenization splits each into a dictionary plus an index list, and those two outputs get their own dedicated processing paths since they benefit from quite different strategies.

Past that point, the main work is over: once data lands in homogeneous streams, OpenZL's default transform heuristics handle the details. Users who want more per-stream fine-tuning can hand the job to the offline trainer.

Automatic Compressor Generation

Fully hand-crafting a compression pipeline is possible, but not required. The simpler route is to describe the data and let the system learn a plan. The workflow has four stages:

1. Describe the input: The Simple Data Description Language (SDDL) maps bytes to fields — rows, columns, enums, nested records. SDDL is parse-only; it just tells OpenZL the shape. Alternatively, a parser function written in a supported language can be registered instead.
2. Learn the config: The offline trainer runs a budgeted search over transforms and parameters, starting from a preset, parser, or SDDL description. It can return a full speed/ratio tradeoff curve or directly target a config that respects given speed limits. Internally it uses a cluster finder to group similar fields and a graph explorer to evaluate candidates.
3. Resolve at encode time: The encoder turns the Plan into a resolved graph. If the plan includes control points, it picks the branch that fits the actual data and records that choice in the frame.
4. Decode without coordination: Every frame chunk carries its own resolved graph. The single decoder validates it, enforces limits, and runs the steps in order. Plan improvements roll out without any new decompressor; old frames decode as before while new data gets the better config.

Adapting When Data Drifts

Real data changes — schema version bumps, content shifts, seasonal spikes. A config frozen at training time would age poorly. OpenZL attacks this on two fronts.

First, periodic retraining. At Meta, where OpenZL extends the Managed Compression system originally built for Zstandard dictionary automation, each registered use case is monitored, sampled, and re-trained. A new config ships when it demonstrably helps, and the decoder never changes.

Second, runtime control points. A compression config can include branches that read lightweight statistics at encode time — string repetition counts, run lengths, histogram skew, delta variance — and pick the best next stage in the plan. The search stays bounded so speed expectations hold, and the chosen branch is recorded in the frame so the decoder just follows the path. This yields dynamic behavior without turning compression into an unbounded search problem, and adds zero decoder complexity.

One Decoder to Rule Them All

The universal decoder is not an implementation detail; it is the core operational advantage. Even when compression configs differ wildly across file types, one binary handles all decompression. That carries real benefits:

• Single audited surface: Security and correctness reviews, fuzzing, and hardening target one binary instead of a growing collection of per-format tools that could drift apart.
• Fleet-wide improvements: Any decoder update — SIMD kernels, memory bounds, scheduling — benefits every compressed file, including files written before the patch.
• Operational clarity: One CLI, one set of metrics, one dashboard. Patching and rollout are routine.
• Continuous training: New plans are trained offline, tested on a sample slice, then rolled out as config changes. Old frames still decode while new frames get improved compression.

How Well It Works — and When It Doesn't

When OpenZL understands the input format, it delivers substantial ratio improvements with fast compression and decompression. The offline trainer also exposes a wide tradeoff space across the three axes of ratio, compression speed, and decompression speed. Instead of the usual compression-level knob, OpenZL configuration is expressed by serializing the compressor graph, which grants far more flexibility in picking the operating point.

The figures below come from structured datasets developed for the OpenZL whitepaper. Reproduction scripts are in the repository, and the raw input data with run logs are published on GitHub. Data points connected by a line are Pareto-optimal: no other point in the same dataset beats them on both metrics simultaneously.

OpenZL has limits. It depends on a description of structure to bring transforms into play. For unstructured content such as plain-text corpora — enwik or dickens, for example — there is nothing to exploit. In those cases OpenZL falls back to Zstandard and performs essentially at parity with it.

Good Candidates for OpenZL

OpenZL is designed for vector, tabular, or tree-structured data, and handles numeric, string, or binary content well. Typical use cases include time-series datasets, ML tensors, and database tables. As with any compression scheme, information theory imposes limits: the input must contain some discoverable order for the tool to exploit.

If your data fits these categories, the OpenZL site and Quick Start guide are the best entry points. Developers can find source code, documentation, and examples in the GitHub repository, where community contributions and feedback are welcome.

Roadmap and Community Opportunities

The project's overall direction is to simplify structure discovery and automate compression plans for evolving data. Immediate development priorities include:

• Extending the transform library for time-series and grid-shaped data.
• Improving codec performance and helping the trainer find better compression plans faster.
• Expanding SDDL to describe nested data formats more flexibly.
• Enhancing the automated compressor explorer to propose safe, testable plan changes within a specified budget.

Community members can contribute on several fronts. For format-specific opportunities, try compressing a structured dataset with an OpenZL prebuilt Plan. If results look promising, generate a new plan with the trainer or customize one using the documentation. Publicly useful formats can be submitted via pull request.

Core contributions are also possible. Developers skilled in C/C++ can help accelerate the engine or add transforms for new data formats. Those focused on reliability can contribute validation rules and resource caps. Benchmarking enthusiasts can add datasets to the test harness so others can reproduce results.

Engagement can start by opening an issue on the GitHub issue board. For use cases where OpenZL underperforms expectations, sharing a few small samples helps the team analyze the problem together. Contributions to codec optimizations, new graphs, parsers, or control points are welcome, as these changes do not affect the decoder's universality.

The project anticipates that OpenZL will expand possibilities in data compression, and the team is interested to see how the open source community applies it.