When Compilers Become Compressors
Facebook’s mobile apps present a scale problem that ordinary compression tools can’t solve: thousands of daily code changes add up quickly, and every added line makes the download larger for billions of users — a real burden in regions where bandwidth is costly. Standard compression helps, but it hasn’t been enough to hold the line on app size. The company’s response is Superpack, a system that pairs classic compression algorithms with compiler analysis to shrink structured data well beyond what tools like Zip or Xz achieve.
Superpack has been used across Facebook’s Android fleet for two years. It targets Dex bytecode (compiled Java), ARM machine code, and Hermes bytecode (Facebook’s JavaScript representation), plus their associated metadata. On those payloads it averages over 20 percent better compression than Android’s default Zip, yielding substantially smaller apps on devices while keeping developer-added size growth in check. Facebook, Instagram, WhatsApp, and Messenger all ship with it.
A Generative View of Data
The project’s foundation is Kolmogorov complexity: the idea that the information content of data equals the length of the shortest program that can reproduce it. If the data you’re compressing is program code in the first place, you can often rewrite it into a smaller program that outputs the original bytes on decompression. That reframing turns compression into code generation, opening the door to decades of compiler research.
Superpack’s novelty is in adopting proven compiler components — parsers, code generators, and even Satisfiability modulo theories (SMT) solvers — for a compression pipeline. In practice, that means the system understands the grammar of the code it’s processing, which allows two standard compression stages to work far better than they could on a raw byte stream.
Sharper LZ Parsing
Lempel-Ziv (LZ) parsing finds repeated byte sequences and replaces later occurrences with pointers to earlier ones. The win comes when the pointer costs fewer bits than the sequence it replaces. Traditional LZ works naively on byte streams, but Superpack groups data along structural lines — for instance, separating the opcode/register part of an instruction from the immediate/address fields. That regrouping has two effects: mathematically identical sequences that were fragmented by unstructured input now sit together, letting the length of matches grow, and distances are measured in logical units along the parse tree rather than raw bytes. Shorter distances mean smaller pointers.
Making these grouping decisions optimal is itself an optimization problem, handled by “hierarchical compression.” In some stretches of code, preserving the original order is better; in others, splitting produces longer matches and cheaper offsets. The optimal-parse stage decides per-segment, and the result is a hybrid stream that mixes untouched spans with split-out, better-compressed groups.
Context-Aware Entropy Coding
When LZ can’t help — the data is unique or too small for a pointer to pay off — literals are written with entropy coding, which assigns short codes to frequent values. Superpack uses an ANS coder but supports pluggable back-end implementations. Here again, structure matters: from its parse of the code, Superpack derives contextual clues that lower the entropy of literals. In a simple branch example, seven distinct addresses all shared the 0x prefix; a naively coded address field would consume about three bits per value. But context reveals that only one opcode (BL or B) pairs with each address — knowing which opcode is present predicts the missing bit, cutting the cost toward two bits. Real gains are fractional, but consistent, because compiler analysis tells the coder which context to use for each literal field.
Turning Data Into Code
Not every payload is structured like a typical instruction stream. Consider Dex references — labels that index into well-known values. These values show high locality, which Superpack exploits by first translating them from a flat list into a miniature program: recent values are “pinned” into logical registers, and subsequent values are emitted as deltas from those pinned entries.
Once this abstraction is in place, the task of keeping references small becomes a register allocation problem — deciding which values stay resident and which get evicted to make room for new ones. The opcodes (MOV, PIN) can then be separated from the deltas and from the reference payload. That split means both improved LZ parsing and lower-entropy literals apply to each resulting stream, reproducing the gains seen in ordinary instruction compression.
The same playbook scales across Superpack’s three payloads. Instruction streams are compressed with a consistent set of transforms driven by knowledge of the target’s syntax. Metadata is handled a second way: first by grouping items of equal types, and then by exploiting specification rules within the metadata itself — sorted orders or the statistical correlation between entries have contextual value that reduces the bits spent on distances and literals.
The resulting ratio, measured for these three formats, is consistently ahead of generic tools:
Superpack is not a different compression theory, but a materially more effective integration: it applies a parser to data, treats the output as a new material, and only then lets an LZ stage and an entropy coder do their work. The outcome is code that becomes smaller than the sum of its parts would otherwise allow, at a time when every megabyte of download matters.
Inside Superpack’s design
Superpack’s architecture is closer to an operating system than to a typical compressor. A central kernel handles paged memory allocation, file and archive abstractions, and instruction transformations, while exposing interfaces for pluggable modules. Atop this kernel sits a compiler layer built around format-specific drivers. Each driver recognizes the properties and label correlations of the data it handles so the compression layer can exploit them. Parsing logic relies on automated inference powered by an SMT solver, which feeds the restructuring engine with fine-grained information about the input.
The compression layer is also modular. Superpack ships with its own compressor—a custom LZ engine with an entropy coding back end—but it can defer to existing tools instead. In that mode, Superpack only reorganizes data into uncorrelated streams, and the external tool performs a best-effort compression. That approach works but is limited by how coarsely the existing tool can leverage compiler-derived information. Superpack’s own back end avoids this ceiling by working at the granularity of individual bits, matching logical correlations in the data’s internal representation. Exposing the compression mechanism as a module lets developers choose between compression ratio and decompression speed.
The implementation is split between OCaml and C. OCaml handles the compression side, where complex compiler-oriented data structures and SMT solver integration dominate. C implements decompression, since that code is simple yet highly sensitive to processor parameters like L1 cache size.
Trade-offs and current boundaries
Superpack is deliberately asymmetric: decompression is fast, but compression may take as long as needed. Streaming compression has never been a goal, and today’s compression speeds cannot keep pace with modern transfer rates in any case. The tool applies to structured data, code, integers, and strings; images, video, and audio are out of scope.
On Android, the biggest tension is between download savings and side effects. Superpack shrinks downloads significantly, but the platform’s delta-update mechanism can only work with archives its diffing tools can interpret. Because those tools cannot recompress Superpack archives, deltas between app versions end up larger. This is an interoperability gap, not a Superpack limitation. Facebook sees a path forward through finer-grained interfaces between Superpack and Android tooling, more flexible distribution mechanisms, and public documentation of the file format. For now, the compression gains are large enough that the trade-off is worth it for users, especially given that Facebook’s apps consist mostly of code types Superpack compresses far better than the compression built into Google Play.
Superpack’s entropy coding back end builds on Jarek Duda’s work on asymmetrical numeral systems. The project also draws on superoptimization research and prior code-compression work. As optional compression back ends, it plugs into Xz, Zstd, and Brotli. Microsoft’s Z3 SMT solver powers the automatic parsing and restructuring across code formats.
Roadmap and future directions
Superpack has already saved substantial download bytes for Android users by compressing Dex bytecode and ARM machine code well beyond what general-purpose tools achieve. Work continues on both the compiler and compression components, and success with diverse data types has broadened the project’s ambitions beyond mobile app size.
Near-term targets include a new on-demand executable format that keeps shared libraries compressed on disk and decompresses them only at load time, plus delta compression of code to shrink software updates. Cold storage is another candidate: Superpack could compress rarely accessed logs and files. Until now deployment has been Android-only, but the techniques apply to other platforms like iOS, and porting work is under consideration. Superpack remains internal to Facebook’s engineers, but the team is exploring ways to improve Android ecosystem compatibility and may consider open sourcing it in the future.
