Snapshot performance: from minutes to milliseconds

Filesystem snapshots in Vercel Sandbox let teams capture and restore an entire sandbox's state. Early versions of the feature were built strictly for reliability—making sure snapshots never failed or lost data. Once that foundation was stable, attention turned to speed. The restore path was painfully slow: p75 restores took over 40 seconds. Through parallelization and local caching, that's now under one second.

What a snapshot actually is

Sandboxes run as isolated containers inside Firecracker microVMs on Vercel's builds infrastructure, Hive. A snapshot is essentially a compressed copy of the sandbox's disk. Two file types are involved:

  • The raw disk image (.img), typically several GBs

  • A compressed version in the custom VHS format (Vercel Hive Snapshot), used for uploads and downloads from S3

sandbox.snapshot() compresses the .img into a .vhs and uploads it to S3. Sandbox.create() with a snapshot does the reverse: downloads the .vhs and decompresses it back to an image. Without compression, each operation would transfer hundreds of MBs to low GBs over the network, adding seconds—sometimes tens of seconds—to every restore.

Parallelize the slow path

The original restore flow was entirely sequential: download the whole .vhs from S3 in a single request, then decompress it using a single thread.

With snapshots ranging from 200MB to a few GBs, the download alone could take seconds to tens of seconds. Using the Range HTTP header, the team switched to parallel chunked downloads, with the AWS Go SDK's transfermanager API handling orchestration. After benchmarking chunk sizes and concurrency levels, download speeds improved 2-5x.

Decompression got the same treatment. The .vhs format stores a header and a frame for each allocated disk region. Instead of decoding frames one by one, a single decoder now feeds N decompression goroutines, speeding up the .vhs-to-.img restore by 2-4x depending on snapshot size.

Even with both stages parallelized, the pipeline still wrote downloaded data to disk before decompression started. Piping S3 range request streams directly into the decompressor removed that intermediate step, cutting end-to-end restore time by another 2x.

There was no fast path

The parallelized pipeline only improved the cold path—when a snapshot had to be fetched from S3 on a cache miss. But there wasn't a cache at all, so every restore was a cold path. Performance was simply not the initial focus.

Sandboxes run on metal instances with NVMe disks, leaving terabytes of fast local storage mostly idle. That storage now hosts a local disk cache with LRU (least recently used) eviction, sized by total disk space rather than entry count. The cache stores the decompressed .img, not the compressed .vhs, so a cache hit skips both download and decompression. When the cache fills, the least recently used snapshots are evicted.

Since most customers reuse a "base" snapshot across many sandboxes, the cache hit rate sits at 95%. On a hit, boot time is essentially just the time to start the microVM and container.

The numbers and what's next

p75 restore latency dropped from 40 seconds to sub-second; p95 went from 50 seconds to 5 seconds. With the cache hit rate, most sandbox boots no longer hit the download-and-decompress pipeline at all.

Further optimization is possible. Cache affinity—routing sandboxes to metal instances that already have the requested snapshot cached—could eliminate the cold path for popular snapshots, but risks thundering herds and hotspotting individual machines. The team is approaching that deliberately. The long-term goal is a fast enough cold path that caching becomes a bonus, not a requirement.

These optimizations already power Automatic Persistence, now in beta, which snapshots a named sandbox's filesystem on stop and restores it on resume. With sub-second restores, that cycle feels instant. Filesystem snapshots are available for all Vercel Sandboxes, with details in the Sandbox documentation.