How we migrated the database behind every Vercel build

Moving the warm pool's state out of Redis

Every Vercel build starts in the build warm pool: a set of standby containers that let builds launch without waiting for new compute. The pool depends on state that tracks which containers are ready, the tokens they use to authenticate, and the mapping from each running build to the deployment that gets billed. When the pool was first built, all of that lived in Redis—fast and sensible at the time.

Over the years, the state became more important than the store holding it. Tokens and container statuses could be rebuilt if lost, but billing mappings could not. Yet all of it sat in a store that was run as an ephemeral cache. That state needed durable storage, which led to a migration to DynamoDB.

The complication: the pool never stops. Containers come up, poll, pick up work, and expire around the clock. There was no window to pause, copy, and restart. The migration had to happen live under production traffic, split into phases, each behind a feature flag with a rollback path.

Why Redis became a liability

Redis was efficient for the pool's original access patterns. But as the pool grew, data that needed to persist was living in structures designed for speed. A lost token could be rebuilt in about ten minutes; a lost mapping meant a build was never billed, because nothing else recorded which deployment it belonged to.

DynamoDB offered what the pool needed: on-demand scaling for bursty deployment traffic, native TTL, and no connection management at high concurrency. What it did not offer was Redis-level latency.

The Redis data layout

• Tokens in a set for membership checks, plus a sorted set for expiration
• Each lifecycle status (pending, polling, building) in its own sorted set—moving a container meant removing from one and adding to another
• A string linking each working container to its deployment

These operations were around a millisecond each, and that speed got baked into the code. One run of the supply loop—the process that refills each warm pool—would make hundreds of count calls just to tally pending and polling containers.

Designing the schema from actual access patterns

The migration plan started by listing everything the code actually asked of the state:

• Verify a token when a container polls for work
• Add a token when a new container comes up
• Expire tokens past their deadline
• Count tokens to size the pool
• Move a container between lifecycle statuses
• Count containers by status
• Look up the deployment behind a container's callback
• Remove expired containers

Almost every operation already knows which container it's touching—only polling starts from a token. So the model put the container at the center. The container ID became the sort key; the token became a field stored as a hash so reads never returned a usable credential. Verifying a token became a lookup of its container and a hash comparison.

Two patterns couldn't be served by key lookup alone. Status counts needed to skip expired containers, requiring a time-aware index, and billing-related deployment lookups needed consistent reads, so that mapping got a small table of its own.

type ContainerRecord = {
  warmPoolId: string   // partition key
  containerId: string  // sort key
  token: string        // stored as a hash
  status: 'pending' | 'polling' | 'building'
  expiresAt: number    // one expiry for the whole record
}

A simplified sketch of the shipped shape: three sorted sets became one field.

Reading state, moving status, and clearing tokens all became direct key lookups. Counting a status was a single read against the time-aware index, with expired containers already filtered out.

// count containers in one status via the index, skipping expired ones
queryCount({
  partition: `${warmPoolId}#${status}`,
  where: 'expiresAt > now',
})

One query per status, bounded by expiry.

DynamoDB forced explicit keys, indexes, conditional writes, and TTL behavior, so rather than port the Redis structures, the team modeled the container and added indexes its access patterns needed.

Shadow mode validated every write

The rollout moved through feature-flagged phases: Redis-only, dual writes, shadow reads, DynamoDB-primary, and finally DynamoDB-only. Each phase kept a rollback open as long as possible. Only the last step, removing Redis writes, lacked an easy rollback—but any lingering token expired within ten minutes anyway.

The process started by baselining existing Redis operations for counts and latencies, giving dashboards a normal to compare against. New DynamoDB methods were merged days before they were called. Dual writes went out with Redis still authoritative and DynamoDB failures logged, not fatal. Shadow reads followed, querying both stores and comparing. Primary reads flipped only after the new path proved itself.

These comparisons checked that both stores agreed on stored values—something tests couldn't verify. They also tracked that every write landed in both places, expiration kept moving forward as containers changed status, clearing a token left the rest of the record intact, and status counts stayed close enough to steer the pool.

Dashboards monitored match rates, write errors, per-query latency, and expiration counts while Redis still served production. Every mismatch was chased down to a bug, a dual-write race, or an expected difference before advancing. The dashboards decided when each phase moved forward.

Two failures along the way

In March, builds in one region went down. The cause traced back to the comparison machinery itself—checking two stores against each other added load that the throttling couldn't contain. The scenario had been flagged in review, and comparisons were throttled for exactly that reason, but not enough. Four days later, a pull request citing the incident added the index status counts needed. Without it, every count was O(n) work, and comparisons were counting constantly.

Later that month, the Redis infrastructure the migration was leaving actually went down. The warm pool and token handling stayed up—both were already reading from DynamoDB as the source of truth. Builds still felt the outage through services further up the pipeline that hadn't moved yet. The failure the migration targeted arrived early, and the state that had already moved survived.

The supply loop choked on slower reads

The real problem surfaced in April when the supply loop began to stall. Shadow data had looked healthy, and per-query latency appeared fine on dashboards. Neither measurement captured what failed.

the loop as designed
  check → create → check → create → check → create → ...
          one state read before every container
what slower reads demand
  check → create, create, create, ... → check
          stop paying a read before every create

Redrawn from the investigation's sketch. Every check on the top line is a round trip to the store.

At P95, getWarmPoolTokenCount measured 1.29ms on Redis and 5.13ms on DynamoDB. That was shorthanded as two to three times slower per query—an underestimate.

Each extra few milliseconds is trivial paid once, but the loop paid them before every container, hundreds of times per run. Runs stretched to minutes at their worst, and the pool couldn't stay ahead of demand. It was an N+1 query pattern that Redis had been fast enough to hide. No one wrote down "this loop requires millisecond reads," but that assumption lived in the design.

Rebuilding the loop around the new latency

DynamoDB would never match Redis's millisecond, so instead of chasing it, the team designed the requirement away.

The first attempt was batching. The latency gap was widest at P90, roughly 17x, so the batch design checked state once per 17 containers. But that constant hard-coded a latency ratio that would drift with load—another unwritten dependency on store speed. Concurrency needed no constant at all.

What shipped instead let the supply loop's calls run concurrently. Warm pools no longer waited behind each other, and each call worked from the last state it saw. Overlapping reads achieved what batching promised: no run serialized on a single read. The worst case was a few extra containers from a stale pool picture, accepted in the pull request. The redesign kept durable storage without compromising build start times.

Diagnosing DynamoDB revealed the loop had also stalled under Redis, sometimes for a minute or more—evidence that existed in telemetry all along. Because the pool sits ahead of demand rather than in a request path, nothing had forced anyone to look until the migration did. The redesign fixed a flaw older than the migration itself.

The real migration was of assumptions

The migration completed in April, with Redis calls removed from warm pool paths. Copying data was the easy part. The hard part was finding assumptions built on Redis's speed and redesigning the loop that depended on them. The state every build relies on now lives in durable storage, and the loop managing it no longer serializes on any single read.

The April retrospective summed it up: "Even a 1ms-to-15ms query time degradation on P90 could bring down our warm pool management logic." No one had written that in February when the migration started.