The most popular option to decompress ZIP files from the Rust programming language is a crate simply named zip — At the time of this writing, it has 48 million downloads. It’s fully-featured, supporting various compression methods, encryption, and even supports writing zip files. However, that’s not the crate everyone uses to read ZIP files. Some applications benefit from using asynchronous I/O, e
AW
Amos Wengerfasterthanli.me
·February 7, 2025·Zip
ZIP Files Are a Minefield, and That’s Why Async Is So Messy
Take a look at how Rust handles the ZIP format and you'll find a familiar split. The most popular crate, simply called zip, has racked up 48 million downloads. It's a fully featured library supporting multiple compression methods, encryption, and even ZIP writing. But it's not the crate everyone reaches for.
Some applications want asynchronous I/O, especially when they're decompressing archives fetched from the network. That's why the uv Python package manager, written in Rust with a single maintainer and far less visibility, doesn't use the zip crate at all. Instead, it relies on async_zip.
It's a common recurring situation in the Rust ecosystem: the same logic re-implemented against sync interfaces and then again against async interfaces. The result is duplicated effort, a fractured ecosystem, and more places for bugs to hide.
Which is a shame, since ZIP turns out to be an old, crufty format full of edge cases that deserve more care than two parallel implementations can give them.
Encoding Is a Crapshoot
Despite the existence of both an ISO standard and the freely available PKWARE APPNOTE, there's still plenty to trip over when you open ZIP files in the wild. At itch.io we dealt with a messy corner of the format that traces straight back to its age: it predates UTF-8 by decades. For years, filename encoding inside a ZIP file was whatever code page your particular system used.
The APPNOTE added documentation for an "extra field" marking file names and comments as UTF-8 in 2007 — a fix that was probably fine in an era where ZIPs moved between offices in the same country on floppy disks. But in the real world you can end up with a Japanese game developer zipping files using Windows Explorer and getting filenames in Shift-JIS, in place of code page 437 with its 1981 hardware heritage.
Since the format only tells you whether a filename is UTF-8 or not, the approach I worked out for the itch.io desktop app to install games from anywhere was to take all the textual content — filenames, comments, everything — and run a statistical analysis. Only when certain byte sequences matched known frequencies did we form a guess that the content might be Shift-JIS, for example.
That produced a list of probabilities and then a decision: take the highest and hope. I don't know of any other tool that bothers with this. If I were doing it again, I might just demand a more modern archive format, rather than trying to fish whatever developers shoved into a file upload dialog.
Directories, Slashes, and Unix Modes
The encoding mess isn't the format's only curdle. ZIP doesn't really distinguish files from directories. A directory is simply a length-0 entry whose path ends with a forward slash. Windows? There's a surprise waiting there too: every Windows API actually supports forward slashes as the path separator.
Microsoft's documentation on file paths explains all this quite thoroughly and is worth reading. And by the way, the APPNOTE is unusually explicit on the topic:
"The path stored MUST NOT contain a drive or device letter, or a leading slash. All slashes MUST be forward slashes '/' as opposed to backwards slashes '\' for compatibility with Amiga and UNIX file systems."
To call out the actual APPNOTE section: that's 4.4.17, the file name definition in the v6.3.10 notes.
Of course, if the archive came from Unix, the entry will carry a mode, and the mode bits tell you if the entry is a directory, regular file, or symlink. Out in the wild, you'll notice that symlinks tend to carry their target as the entire contents of the entry, even though the APPNOTE goes another way: it points at the Unix extra field's variable-size data section as the "correct" place to store a symlink or hard link target. And note the wording: can be used.
The reason for that mismatch is that so many tools created ZIPs for so long without any real authority stepping in. ISO standardization only came later and mandated UTF-8 filenames, but the APPNOTE itself is descriptive rather than prescriptive. It documents zip implementations' actual behavior, without judging what those implementations chose.
All of this means a faithful ZIP reader needs to understand a lot of historical mess, starting with timestamps. DOS timestamps fit into 32 bits, a neat dozen for the date and half for the time, and each half is not quite what you'd expect. The date stores a 5-bit day, a 4-bit month, and a 7-bit year counting from 1980. Time is kept in two-second intervals.
Next time you hear someone slam IEEE 754 floating point as "weird" for whatever it does with 0.1 + 0.2, think of the DOS timestamp format and acknowledge it's really all relative.
Start at the End
If those were the only challenges, you could probably pass on some details for files made by newer software. But even the most fundamental layout of the format is slightly cursed.
Nearly every file format on earth starts with a magic number followed by a header full of metadata and then, finally, the payload. ZIP is not one of those.
fasterthanli.me/content/img on main [!?]❯ hexyl logo-round-2.png | head
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
│00000000│ 89 50 4e 47 0d 0a 1a 0a ┊ 00 00 00 0d 49 48 44 52 │×PNG__•_┊⋄⋄⋄_IHDR│
│00000010│ 00 00 01 00 00 00 01 00 ┊ 08 06 00 00 00 5c 72 a8 │⋄⋄•⋄⋄⋄•⋄┊••⋄⋄⋄\r×│
│00000020│ 66 00 00 2a b5 7a 54 58 ┊ 74 52 61 77 20 70 72 6f │f⋄⋄*×zTX┊tRaw pro│
│00000030│ 66 69 6c 65 20 74 79 70 ┊ 65 20 65 78 69 66 00 00 │file typ┊e exif⋄⋄│
│00000040│ 78 da a5 9c 6b 76 5d b7 ┊ 8e 84 ff 73 14 77 08 7c │x×××kv]×┊×××s•w•|│
│00000050│ 93 18 0e 9f 6b f5 0c 7a ┊ f8 fd 15 8f e4 eb 38 ce │ו•×k×_z┊×ו×××8×│
│00000060│ ed a4 db 89 25 59 3a da ┊ 9b 9b 00 0a 55 00 78 dc │××××%Y:×┊××⋄_U⋄x×│
│00000070│ f9 ef ff ba ee 5f ff fa ┊ 57 f0 3e 54 97 4b eb d5 │×××××_××┊W×>T×K××│
│00000080│ 6a f5 fc c9 96 2d 0e be ┊ e8 fe f3 67 bc 8f c1 e7 │j××××-•×┊×××g××××│
The one correct way to read a ZIP file is to open it at the very end and walk toward the back until you find the signature that marks the end of the central directory record — known shorthand among ZIP programmers as EOCD.
That backwards-looking design explains the zip crate's API requirement that anything you hand it implement both Read and Seek. The library can't even enumerate entries without repositioning inside the file.
impl<R:Read + Seek>ZipArchive<R>{/// Read a ZIP archive, collecting the files it contains.
///
/// This uses the central directory record of the ZIP file, and ignores local file headers.
pubfnnew(reader:R) -> ZipResult<ZipArchive<R>>{// ✂️}}
Finding that record turns out to be an actual algorithmic problem. The original zip crate just started 4 bytes from EOF and moved one byte at a time on each failed signature comparison, a staggeringly wasteful process. The async_zip crate improved it significantly, reading chunks of 2 KiB and stepping left by 2 KiB minus the length of the signature to handle signatures splitting across buffer boundaries. Its own comments cite a 500x speedup over the old approach.
Eventually, in May 2024, the zip crate caught up and began doing 512-byte reads. That gave it a big temporary speedup until August, when a bug fix in the EOCD search logic landed — one that's worth a closer look just for the amusement factor.
Framing and the fuzziness at file boundaries
File formats generally provide a way to delineate their own structure. MP4 files, for instance, are composed of boxes, each prefixed by its length and a type. Even if a media player doesn’t recognize a particular box type, it can skip over it safely, confident that the box's type is never conflated with the actual data payload. This absolute separation between metadata and content makes parsing unambiguous.
ZIP files are not so cleanly defined. Their structure is anchored at the end of the file: you locate the End of Central Directory (EOCD) record by scanning backward for its specific signature. The problem? No such clean separation exists for arbitrary data. A comment or a file path could contain bytes that, when read backward, happen to match the EOCD signature. This ambiguity was the root of a bug fixed in the zip crate in August 2024. The fix involved scanning the entire file to catalog every possible EOCD offset, rather than stopping at the first candidate — a strategy that works but can be catastrophic for performance.
Consider a multi-gigabyte file. Scanning it in reverse with 512-byte seeks backward is one of the worst possible I/O patterns for both userland buffering and kernel read-ahead. One user on a 200GB networked share found the fix's cost prohibitive. A subsequent PR, merged in December 2024 after eleven weeks of discussion, rewrote the EOCD detection algorithm again to address the performance regression introduced earlier. This saga highlights how the messy reality of file formats — where a self-extracting ZIP can be prefixed with an entire executable (signaled by MZ) before the actual ZIP payload (signaled by PK) — constantly complicates seemingly straightforward parsing logic.
~/Downloads❯ unzip -l winzip76-downwz.exe | head
Archive: winzip76-downwz.exe
warning [winzip76-downwz.exe]: 2785280 extra bytes at beginning or within zipfile
(attempting to process anyway)
Length Date Time Name
--------- ---------- ----- ----
2700 09-06-2024 18:34 common/css/common.css
21825 09-06-2024 18:34 common/css/jquery-ui.css
30945 09-06-2024 18:34 common/img/arrow.png
14982 09-06-2024 18:34 common/img/button-hover.png
14982 09-06-2024 18:34 common/img/button-normal.png
728365 09-06-2024 18:34 common/img/centerImg.png
17027 09-06-2024 18:34 common/img/close-hover.png
~/Downloads❯ hexyl --skip 2785280 --length 64 winzip76-downwz.exe
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
│002a8000│ 50 4b 03 04 14 00 00 00 ┊ 08 00 51 94 26 59 ad 3a │PK•••⋄⋄⋄┊•⋄Q×&Y×:│
│002a8010│ 80 57 d1 02 00 00 8c 0a ┊ 00 00 15 00 00 00 63 6f │×Wו⋄⋄×_┊⋄⋄•⋄⋄⋄co│
│002a8020│ 6d 6d 6f 6e 2f 63 73 73 ┊ 2f 63 6f 6d 6d 6f 6e 2e │mmon/css┊/common.│
│002a8030│ 63 73 73 b5 56 cb 6e db ┊ 30 10 bc 07 c8 3f 10 30 │css×V×n×┊0•×•×?•0│
└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘
This complexity is precisely why the rc-zip crate was designed differently. Instead of tying itself to a particular I/O style, it adopts a "sans-io" approach, decoupling the parsing state machine entirely from the mechanics of reading data. This pattern is well-established — the rustls crate uses it, and it is pervasive in C libraries, which have no standardized I/O interface. The ZStandard decompression API, for example, uses a raw buffer struct — pos and size fields — and lets the caller drive the process, inferring the state from the positions. This clear protocol avoids ambiguity and gives the caller complete control. The rc-zip crate applies this same principle, but its task is harder because it must first scan backward from the file's end to read the central directory, and then allow random access to individual entries.
Two state machines, one clean loop
To handle this, rc-zip exposes two separate state machines. ArchiveFsm reads the central directory, and, upon completion, yields an Archive. From the archive, you can construct an EntryFsm for each entry, which knows its offset and compression method. Driving these machines is a straightforward loop: first, call wants_read. If the machine needs more data, it returns Some with the required file offset. Next, borrow the internal buffer with space, which provides a mutable slice of the maximum data you can read. After performing the read, call fill with the byte count, with a zero read denoting end-of-file.
Finally, call process, a method that consumes the state machine. If it's finished, it returns the Done variant of FsmResult, and the machine is gone, preventing any accidental re-use. If it needs more data, it returns the Continue variant, which yields ownership back to the caller for another loop iteration. This design is intentionally pragmatic; while one could go deeper with typestates, this approach integrates simply with both synchronous I/O via rc-zip-sync and asynchronous I/O via rc-zip-tokio.
The async file I/O quagmire
But as with all things in Rust, the devil is in the details of the runtime. The rc-zip-tokio implementation is messy for a good reason: asynchronous file I/O on Linux is messy. Tokio doesn’t have a magical async file read; it uses a background thread pool to perform blocking reads and wakes up the future when data is ready. This is not tokio’s fault — for a long time, there was simply no elegant way to perform true async file reads on Linux.
Consider the difference in just reading one GiB of data. On a Linux server, using libstd’s synchronous read is consistently faster than using tokio. Why? An strace-like tool, lurk, reveals the two approaches' very different natures over the program's course. The async version performs a repetitive dance: a thread reads 128KiB, which wakes up another thread to queue more work, and so on — eight thousand times. The synchronous version does this instead — a single, majestic 1GiB read syscall.
The game changed with io_uring. A tokio-uring variant remains consistently competitive with the sync version and about 10% faster than the classic tokio approach, even when the test program is constrained to small reads. The read loop for the tokio-uring version shows the new dynamic: it submits the operation via io_uring_enter, waits for completion via epoll_wait, and then uses a write call to a task event to wake itself up. That write call triggers tokio's internal channel-based waker mechanism, which relies on an eventfd descriptor under the hood.
This mixing of epoll and io_uring is an overhead that some developers choose to avoid entirely by creating their own runtimes. The glommio, monoio, and nuclei projects are examples. A monoio-based benchmark of the same read test shows the hot loop reduced to just io_uring_enter, eliminating the intermediary epoll_wait and wake-up syscalls entirely.
It's essential to note this is not a formal benchmark; it was merely an observation of how different systems behave. But the implications are clear for a library like rc-zip: by isolating the parsing logic from the I/O, it becomes trivially portable to a new runtime like monoio or a raw io_uring interface, without any changes to the core parsing logic. The design allows the library to adapt to the best available I/O strategy on any given platform.
An rc-zip-monoio proof of concept
To show what a sans-io design enables, let's wire rc-zip into monoio. The goal is a single async function that takes a file reference and returns an Archive or an error.
pubasyncfnread_zip_from_file(file:&File) -> Result<Archive,Error>{// TODO: the rest of the owl}
The monoio file type provides a native read_at method, but its signature differs from the usual Tokio conventions:
Instead of borrowing a buffer, this method takes ownership of it and returns it — even on failure. This is what a memory-safe io_uring interface in Rust requires: the buffer can't be freed before the kernel finishes with it, since ownership is effectively transferred to the kernel for the duration of the operation.
That API shape makes the code a bit unusual. First, the buffer isn't a Vec<u8>; there's no need to track capacity separately or allow growth. A boxed slice of 256 KiB, fully initialized, is sufficient:
After querying the file size, the state machine is created and the loop begins:
let meta = file.metadata().await?;let size = meta.len();letmut fsm = ArchiveFsm::new(size);loop{// rest of the code goes here}
Whenever the machine requests a read:
ifletSome(offset) = fsm.wants_read(){// rest of the code goes here}
…the first step is calculating the largest read that fits. The machine's buffer can't be handed to the kernel because the current rc-zip API only lends it mutably — it doesn't transfer ownership. Data must be read into our own buffer first and then copied into the machine's buffer.
Changing that API would be straightforward but breaking, so it's deferred to a future release.
The read size is the minimum of both buffer sizes:
Whether the operation succeeded or failed, the buffer comes back. Errors are propagated first, then the bytes read are copied into the machine's buffer via its fill method:
let n = res?;(dst[..n]).copy_from_slice(&slice[..n]);
fsm.fill(n);
Ownership of the buffer is reclaimed from the SliceMut returned by read_at:
buf = slice.into_inner();
That's why buf is a mutable binding: it's moved out each loop iteration on the condition that it's put back. The Rust compiler enforces this strictly:
error[E0382]: borrow of moved value: `buf`--> rc-zip-monoio/src/lib.rs:35:42
|27 | let mut buf = vec![0u8; 256 * 1024].into_boxed_slice();
| ------- move occurs because `buf` has type `Box<[u8]>`, which does not implement the `Copy` trait...30 | loop {
| ---- inside of this loop...35 | let max_read = dst.len().min(buf.len());
| ^^^ value borrowed here after move...41 | let slice = IoBufMut::slice_mut(buf, 0..max_read);
| ------------------------------------- `buf` moved due to this method call, in previous iteration of loop|note: `slice_mut` takes ownership of the receiver `self`, which moves `buf`
--> /Users/amos/.cargo/registry/src/index.crates.io-6f17d22bba15001f/monoio-0.2.4/src/buf/io_buf.rs:256:22
|256 | fn slice_mut(mut self, range: impl ops::RangeBounds<usize>) -> SliceMut<Self>
| ^^^^help: you can `clone` the value and consume it, but this might not be your desired behavior
|41 | let slice = IoBufMut::slice_mut(buf.clone(), 0..max_read);
| ++++++++
Finally, process is called on the state machine, either breaking the loop or continuing:
This runs on macOS using monoio's legacy driver, and on Linux with the io-uring driver. From the io_uring_setup call onward, no read or write syscalls appear in the trace — everything happens as io-uring operations:
The only visible syscalls are brk and mmap-related, which are tied to heap allocation.
The EntryFsm state machine is left as an exercise; a draft pull request on the rc-zip repository shows one approach. It's simpler in some ways since reads are linear, but more involved because it streams data out during decompression. Either way, it only needs to be written once to unlock every compression method rc-zip supports — deflate, bzip2, LZMA, and ZStandard.
The sans-io path forward
Alternatives like keyword generics aim to bridge the sync/async divide, but implementing formats and protocols in a sans-io style is the more direct route. Unifying libstd and Tokio would be solving the wrong problem: neither interface is compatible with modern I/O APIs like io_uring.
This isn't a purely theoretical position. The HTTP implementation loona is tied to a specific I/O model, a choice made while still learning HTTP internals. In hindsight, a fully sans-io rewrite would let it serve multiple contexts: high-performance proxies with monoio, web applications on classic Tokio, and synchronous CLI tools that don't need async at all.
The rc-zip interface also deserves another look — eliminating the copy between I/O buffers and decoding buffers will require rethinking parts of the API to be uring-friendly.
It's worth noting that ecosystems without a standard I/O abstraction, like C, or with much higher-level ones, like Node.js, adopted io_uring faster than Rust did, where substantial code was written against a less flexible model. That's a fair criticism of the Rust ecosystem, not a dismissal of it.
Use production traffic and security signals to prioritize findings, prepare edge mitigations when safe, and propose code patches. By combining WAF data with OpenAI Daybreak models, Vulnerability Discovery and Remediation helps teams identify and patch the most critical threats first.
KSKen Sanderson, Jacob Crisp·September 3, 2026Security
Bot operators have historically had the economic advantage, bypassing static, deterministic detection rules with cheap proxies and retooling. Cloudflare's new Adaptive Intelligence engine flips this dynamic by autonomously learning from the meta-signals of live traffic and deploying disposable rules, making automated attacks too expensive to sustain.
