Why Idempotency Keys Matter

An idempotent API endpoint can be called any number of times with the same result: the intended side effects happen only once. That guarantee becomes invaluable when clients face ambiguous failures, like a dropped connection or a server crash mid-request. Instead of risking a duplicate charge or a double booking, a client can simply retry the same request until it receives a definitive answer.

The simplest way to implement idempotency is with a local ACID database. By mapping each request to a transaction, you can make the whole endpoint safe with far less machinery than what follows. If that's enough for your service, take that path.

Problems start when an endpoint has to reach outside its own database. Charging a customer through Stripe, sending a receipt via email, writing a DNS record — these are "foreign state mutations." Once you commit an action in another system, you can't roll it back. The answer in that scenario is an idempotency key, a concept popularized by Stripe's API.

A client generates a unique key, attaches it to a request, and includes the same key on every retry. The server stores the key with the request's state and uses it to recover from a failure. Once the request definitively finishes — whether by success or a terminal error — the server caches the final result and returns it to any later request carrying the same key. Keys aren't archives; you should recycle them after a horizon of about 24 hours.

The key is often transmitted as an HTTP header:

POST /v1/charges

...
Idempotency-Key: 0ccb7813-e63d-4377-93c5-476cb93038f3
...

amount=1000&currency=usd

Designing a Reference Implementation

To illustrate the mechanics, let's build out a server for Rocket Rides, Stripe's sample ride-hailing app. A ride request would follow this lifecycle:

  1. Insert an idempotency key record.
  2. Create a ride record to track the trip.
  3. Create an audit record referencing the ride.
  4. Call Stripe to charge the user — an external call with real risk of failure.
  5. Update the ride record with the charge ID from Stripe.
  6. Email the user a receipt.
  7. Update the idempotency key with the results.
A typical API request to our embellished Rocket Rides backend.
A typical API request to our embellished Rocket Rides backend.

When everything goes smoothly, this sequence is trivial. At thousands of calls per day, occasional failures will creep in — a constraint violation, a database blip, a Stripe call that times out, a Mailgun delivery failure. At millions of calls per day, these errors become routine. The client may retry a request whose earlier attempt actually succeeded, or a mobile user may drift offline in the middle of a ride booking. The backend has to stay safe through every scenario.

A Symphony of Failure Modes

A typical implementation might naively chain these steps together, but each stage introduces potential failure. Even "internal" calls count as foreign state mutations. Pushing a record to Kafka feels safe because it rarely fails — treat it like any other fallible external dependency.

The distinction between handling local and foreign state is critical. Local work within an ACID database can be rolled back cleanly. But the moment you make a foreign call, you're committed. Your own Postgres instance can't undo a charge that Stripe processed.

Designing Atomic Phases

The bookkeeping between foreign calls can be split into atomic phases. Each phase is a set of local state mutations that either completes entirely inside a transaction or rolls back to nothing. Crucially, each phase must be committed before you initiate the next foreign call. That leaves a durable record so a later retry knows what did or didn't happen.

A recovery point marks the furthest step the attempt reached. The recovery point name is stored right on the idempotency key record, letting a manual review reveal exactly how far an operation got before breaking down. A freshly-triggered request gets the started mark; when the request state transitions, the new recovery point should follow the same transactional pattern: update and swap atomically.

You can't always defer work to background jobs, but you should when you can. In-band foreign calls slow down the request and sprawl your failure modes. Prep work like sending a JWT or creating a customer profile usually can be put on a queue without affecting the user's flow.

Idempotency Beyond APIs

The same mechanics apply to far more than HTTP APIs. Anywhere two systems exchange data subject to pauses and restarts should interoperate through immutable, idempotent operations. Failing to do so introduces subtle bugs that only surface when a partial failure sneaks through — the exact kind of bug that's hardest to trace in production.

Designing systems this way is fundamental rather than cosmetic. Each atomic phase should assert its own assumptions rather than leaving them implicit. Operations written with recovery in mind become much harder to misuse, because the application expresses the boundaries it expects rather than leaving you to reconstruct them after an unexpected outage and a half-finished write.

Putting the schema together

Before building the actual phases, Rocket Rides needs a table to track idempotency keys. The core fields are:

  • idempotency_key: The user-supplied key, with a length constraint to prevent abuse. It is unique per (user_id, idempotency_key) pair, so different users can reuse the same key.
  • locked_at: Marks the key as being actively worked. The first request locks it; concurrent retries also set this field to ensure only one request proceeds.
  • params: Stores the original request inputs, mainly to detect when a client sends the same key with different parameters. It also enables background processes to push unfinished requests to completion.
  • recovery_point: A text label for the last completed phase. Starts as started and becomes finished when the request is done.

Around this, the rest of the app uses standard database best practices wherever possible: NOT NULL, unique constraints, and foreign keys on the relations for rides, users, and audit records.

Defining atomic phases

The request lifecycle is split into distinct atomic phases by following three rules:

  1. The idempotency key upsert gets its own phase.
  2. Every mutating call to a foreign system (Stripe, an email provider, etc.) gets its own phase.
  3. All remaining operations between those phases are grouped together — any number of ACID database operations can safely share one phase.

For Rocket Rides, this yields four phases: tx1 for the key insertion, tx2 for local record creation, tx3 for the Stripe call, and tx4 for finishing up. Each phase can be resumed from the recovery point the prior committed phase set: started, ride_created, or charge_created.

Each atomic phase runs inside a transaction and can return one of three outcomes:

  1. A RecoveryPoint, which sets a new recovery point in the same transaction and continues to the next phase.
  2. A Response, which marks the key as finished and returns a result. Use this for normal success or for errors that can never succeed on retry, such as a declined credit card.
  3. A NoOp, which continues execution without changing recovery point or response.

Serialization errors surface as 409 Conflict, since they almost always indicate a concurrent request interfered. In practice, you would retry immediately rather than surface the error, as the next attempt is likely to succeed. Every other error returns 500 Internal Server Error. In both cases, the idempotency key is unlocked before returning so that another request can pick the work back up.

Locking an idempotency key

When a new key arrives, the API either inserts a fresh row or locks an existing one. An already-finished key is not an error — it simply falls through to the standard success path and returns the stored response. If the key is already locked by another in-flight request, the client gets a 409 Conflict.

Concurrent attempts to lock the same key are not a concern, even without an explicit SELECT ... FOR UPDATE, because the phase runs in a SERIALIZABLE transaction: Postgres aborts one of the two conflicting transactions automatically.

Stepping through the phases

The flow is a directed acyclic graph rather than a mutable loop. Each phase is entered from a recovery point that was either read off a recovered key or set by the prior phase. The loop exits at finished; an already-finished key breaks the loop immediately and returns its stored response.

The second phase, tx2, is simple bookkeeping: insert a ride row, write an audit record, and set the recovery point to ride_created.

The Stripe mutation in tx3 attempts a $20 charge against the customer ID stored on the user record. On success, the ride created by tx2 is updated with Stripe's charge ID and the recovery point becomes charge_created. Unrecoverable payment errors — a bad card, a decline — set the key to finished and respond to the client right away, as retries can never change that outcome.

The final phase sends the receipt. Because the mail job is staged in a transactional job drain, the enqueue commits atomically with the phase’s other writes. The last step records success and returns the response.

Supporting processes

Three background workers round out the architecture. The enqueuer moves staged jobs into the real queue only after their originating transaction commits.

The completer exists for a specific failure mode: clients that give up on an indeterminate request and never retry. Its only job is to find abandoned keys that never reached finished and push them through, using an internal authentication path to retry on any user’s behalf.

The reaper eventually deletes old idempotency keys entirely. A retention window around 72 hours is a reasonable default: long enough to cover a Friday bug deployment that fails requests through the weekend, short enough that the table never becomes an archive. An improved reaper might also flag unprocessable requests for a human to review.

Failure scenarios, end to end

With all pieces in place, consider what actually goes wrong:

  • A dropped request before it reaches the backend: The client retries with the same key and succeeds.
  • Two simultaneous attempts at the same key: The unique constraint lets exactly one through; the other receives 409 Conflict.
  • The database goes down after the key is created: The client keeps retrying, and once the backend returns, the key is recovered and the request continues from its stored recovery point.
  • Stripe is unavailable: The atomic phase fails, the client is told to retry, and nothing succeeds until Stripe is healthy again.
  • The server dies mid-Stripe-call: Because the API’s call to Stripe uses Stripe’s own idempotency mechanism, a retry with the same key cannot double-charge.
  • A bad deploy 500s every request: After a fix is shipped, client retries complete along a clean path. If clients vanish, the completer finishes the work.

Each failure is contained by the recovery points and atomic phases; none of them can leave the system in a state where a retry produces an incorrect or duplicated result.

Handling Unsafe Foreign Calls

When a foreign API supports idempotency keys or is inherently idempotent, retrying a failed call is safe. Most services, however, do not make that guarantee. If a non-idempotent operation fails, you typically cannot know whether the remote side committed the change before the error occurred. The conservative choice is to mark the operation as permanently failed.

The only safe retry case is when the error itself explicitly indicates the operation can be retried. Indeterminate failures like a connection reset or a timeout give you no such assurance and must be treated as terminal failures. This asymmetry is why implementing idempotency keys on your own services matters: it converts ambiguous failures into retryable ones.

The ACID Requirement

All of these guarantees rely on transactional semantics. On a non-ACID store like MongoDB, none of this is achievable. Without atomic commits, every write is effectively a foreign state mutation from the perspective of your application logic—there is no atomic phase to serve as a safe recovery point.

Reuse Beyond APIs

The same idempotency-key pattern applies outside of HTTP APIs. Double form submission is a classic web application problem: a user clicks “Submit” twice, generating two separate HTTP requests. If the submission has non-idempotent side effects—such as charging a card—that is a bug.

You can prevent this by emitting a hidden <input type="hidden"> containing an idempotency key when the form is first rendered. The value stays constant across repeated submissions, and the server-side handler deduplicates requests against that key.

Designing for Passive Safety

The goal for any API backend should be what we call passive safety: regardless of the failures thrown at it, the system converges to a stable state without manual intervention. Once that is guaranteed, active mechanisms (retry queues, completion workers, reapers) can push the system toward full consistency.

Purely idempotent transactions and the idempotency-key-plus-atomic-phase pattern described here are two movements in that direction. Failures are treated as inevitable, and the design accounts for them explicitly. The result is a system that tolerates downtime, partial writes, and ambiguous errors cleanly.

One theoretical alternative is two-phase commit between your system and every other service you call, which would allow distributed rollbacks. It is rarely seen in practice because the complexity and implementation cost across disparate systems are prohibitive.