Durable Objects: Easy, Fast, Correct — Choose three

The Hard Parts of Durable Object Storage

Distributed storage is one of those problems that looks simple on the surface and turns out to be anything but. For application developers, the trouble usually isn't in the database or consensus layer—it's in the code that talks to those systems. Subtle timing bugs can produce inconsistent results or, worse, silent data loss, and these failures often only surface under heavy load or after a machine dies unexpectedly.

Durable Objects were no exception. Each Durable Object is a Cloudflare Worker with its own private, persistent key/value storage, running in a single location on a single thread. That design is meant to make in-memory state synchronization trivial: a given piece of data belongs to exactly one thread at any moment, so there are no multi-client races to reason about. But storage access is still I/O, and that means Promises, await, and all the concurrency hazards those bring back into an otherwise single-threaded world.

Rather than continue asking developers to navigate those hazards carefully, Cloudflare changed the runtime. As of last month, many Durable Object applications that previously contained latent race conditions are correct by default, and many that were slow are now fast—with no code changes required. The promise is that intuitive code now works as written.

Single-Threaded, But Not Race-Free

The fundamental appeal of Durable Objects is their strict execution model. Each object runs in exactly one place, one thread at a time, and can keep state and do synchronization in memory without worrying about other clients touching the same bytes. That's the killer feature. But the storage API is asynchronous: every get() and put() returns a Promise that must be awaited. And every await is a point where execution can pause and let other code run.

Consider a simple counter implementation:

// Used to be slow and racy -- but not anymore!
async function getUniqueNumber() {
  let val = await this.storage.get("counter");
  await this.storage.put("counter", val + 1);
  return val;
}

That code looks like it should return a fresh, monotonically increasing number on every call. It doesn't. The problem is that two concurrent requests can interleave at the awaits. If request A calls get("counter"), then yields, request B can also call get("counter") before A has done its put("counter", val + 1). Both reads see the same value, both write val + 1, and both return the same number.

The interleaving isn't just a theoretical concern—it's the sort of bug that's nearly impossible to test for. It only manifests when multiple requests hit the same object around the same time, and even then only occasionally. Under light traffic, everything looks fine. Under a spike, unique identifiers start duplicating.

That correctness issue was compounded by a performance problem. Each call needs two storage round trips: a get() and a put(). The read takes a few milliseconds. The write takes much longer, because await put() must not return until the data is durably stored. That means writing to multiple disks on multiple machines, and replicating across multiple Cloudflare locations—all of which takes tens of milliseconds, governed by the speed of light. A function that does one read and one write can easily consume 50–100ms per call, and an application that calls it in sequence gets slow in a hurry.

Why Not Just Fix the App?

There are two standard remedies for this kind of problem, and both have serious drawbacks.

Transactions: Correct, But Ugly

The Durable Objects storage API has always supported transactions, and they do fix the race condition:

// No more race condition... but slow and complicated.
async function getUniqueNumber() {
  let val;
  await this.storage.transaction(async (txn) => {
    val = await txn.get("counter");
    await txn.put("counter", val + 1);
  });
  return val;
}

With a transaction, if two calls interleave, the system detects the conflict, lets one complete, and retries the others so they see the winner's write. That's correct, but it comes at a price. Transactions add coordination overhead, making the operation slower than the naive version. And retries become more likely under high load—exactly when you can least afford them.

The bigger problem is that retries are invisible to most developers. The transaction callback can be invoked multiple times, and it's remarkably easy to write a callback that isn't idempotent—especially if it mutates in-memory state alongside on-disk state. Standard tests rarely catch this because retries only happen on conflict. The result is a foot-gun that many developers will eventually fire.

In-Memory Caching: Fast, But Fragile

Durable Objects' real superpower is that only one instance exists at a time, so you can safely cache state in memory instead of reading it from disk on every call:

// Much faster! But (used to be) wrong.
async function getUniqueNumber() {
  if (this.val === undefined) {
    this.val = await this.storage.get("counter");
  }

  let result = this.val;
  ++this.val;
  this.storage.put("counter", this.val);
  return result;
}

That version is dramatically faster. After the first call warms the cache, subsequent calls return immediately with no I/O and no opportunity for concurrency—so they always return unique numbers. That's both faster and more correct than the original.

But it introduces two new problems. First, there's still a race on initialization. If two requests arrive before the counter has ever been read, both can initialize the cache, and one will clobber the other. Making initialization safe requires a shared initialization Promise and careful error handling, which is surprisingly tricky to get right. Second, because the put() isn't awaited, it can be silently lost. If the machine hosting the object dies before the write completes, the object restarts elsewhere with stale data and starts reissuing numbers that were already handed out. Awaiting the put() fixes that, but then the function is slow again—and slow code creates more opportunities for race conditions in the calling code.

Making concurrency safe by default

Faced with these challenges, we had two possible paths: document the pitfalls extensively and hope developers write correct code, or change the system so naturally-written code is correct and fast by default. We chose the latter, implementing it in three parts.

Input gates: deferring events during storage operations

Can our original example work correctly under concurrent requests? Yes, with a simple rule:

Input gates: While a storage operation is executing, no events shall be delivered to the object except for storage completion events. Any other events will be deferred until such a time as the object is no longer executing JavaScript code and is no longer waiting for any storage operations. We say that these events are waiting for the "input gate" to open.

// Can this "just work" please?
async function getUniqueNumber() {
  let val = await this.storage.get("counter");
  await this.storage.put("counter", val + 1);
  return val;
}

With this rule, storage operations no longer create an opportunity for concurrency. Concurrent requests get serialized at the input gate, so each call to getUniqueNumber() returns a unique number even under load.

The rule doesn't forbid concurrent storage operations themselves. You can still issue multiple storage calls without awaiting each one:

let promise1 = this.storage.get("foo");
let promise2 = this.storage.put("bar", 123);
await promise1;
frob();
await promise2;

Here, get() and put() run concurrently, and frob() may execute before the put() completes (though strictly after the get(), since we awaited that promise). Crucially, no other event can unexpectedly interleave — such as a new incoming request.

The protection extends beyond incoming requests. Consider concurrent responses to outgoing requests:

async function task1() {
  await fetch("https://example.com/api1");
  return await this.getUniqueNumber();
}
async function task2() {
  await fetch("https://example.com/api2");
  return await this.getUniqueNumber();
}
let promise1 = task1();
let promise2 = task2();
let val1 = await promise1;
let val2 = await promise2;

This launches two fetch() calls concurrently, with getUniqueNumber() invoked after each completes. These calls cannot interfere. A fetch() completion is itself an event, and input gates defer such events while storage operations are in progress. If the first fetch returns and starts storage operations, and the second fetch returns while those operations are pending, the second return waits until the storage work finishes.

There is a caveat, which async experts will spot. If both calls are initiated from the same event:

// Still a problem even with input gates.
let promise1 = getUniqueNumber();
let promise2 = getUniqueNumber();
let val1 = await promise1;
let val2 = await promise2;

…then no event exists that can be deferred between the two calls. The application fails to await the first getUniqueNumber() before starting the second, so they run concurrently and can interfere. The system cannot distinguish this from code that legitimately intends parallel storage operations. However, this bug is deterministic rather than dependent on unpredictable network timing — far easier to reproduce and catch in testing. We consider this an acceptable trade-off.

Output gates: holding outgoing messages for write confirmation

The in-memory caching example had two problems: the initialization race (solved by input gates) and the choice between awaiting put() (slow) or not awaiting it (risking data loss). A second rule addresses the latter:

Output gates: When a storage write operation is in progress, any new outgoing network messages will be held back until the write has completed. We say that these messages are waiting for the "output gate" to open. If the write ultimately fails, the outgoing network messages will be discarded and replaced with errors, while the Durable Object will be shut down and restarted from scratch.

With output gates, you no longer need to await put(). Code can proceed on the assumption the write will succeed. If it doesn't, nothing the application does afterward becomes observable anyway. A premature success response to a user won't be delivered until the put() completes — so by the time the user receives it, the confirmation is accurate. On the rare write failure, the message is never sent at all.

Output gates apply not only to responses to clients but also to new outgoing fetch() requests. Those are delayed until all prior writes are confirmed, making it impossible for any external party to observe a premature confirmation.

With this rule, the in-memory caching getUniqueNumber() is fully correct while retaining most of its speed advantage. Except for the first call, the application never blocks waiting for the operation. The final response is delayed only pending write confirmation, which can overlap with any subsequent writes.

Automatic caching in the storage layer

The manual caching pattern works but is awkward. The classic solution — used by operating systems for disk storage — is to add caching directly to the storage layer.

Durable Objects now include an in-memory caching layer, keeping up to several megabytes of data in the process where the object runs. A get() for a cached key returns immediately without even context-switching out of the object's thread and isolate. Misses still require a storage request, but reads complete relatively quickly.

put() requests now complete "instantaneously" by writing to cache. Output gates prevent premature external confirmation of writes, and writes are coalesced so the gate waits only O(1) network round trips, not O(n) — even when you await them.

Because get() and put() now complete instantly in most or all cases, input gates have far less negative impact on throughput — the gate spends little time blocked.

With built-in caching, the simple code is just as fast as the manually-optimized version. Combined with input and output gates, code is simple, fast, and correct simultaneously.

Bonus consistency guarantees

The caching layer adds two consistency guarantees beyond performance.

First, writes are automatically coalesced. Multiple put() or delete() operations issued without awaiting in between are grouped and stored atomically. After a sudden power failure, either all writes survive or none do:

// Move a value from "foo" to "bar".
let val = await this.storage.get("foo");

this.storage.delete("foo");
this.storage.put("bar", val);
// There's no possibility of data loss, because the delete() and the
// following put() are automatically coalesced into one atomic
// operation. This is true as long as you do not `await` anything
// in between.

Second, read ordering is now deterministic. Previously, overlapping storage operations had no guaranteed order — a get() and put() on the same key, initiated without awaiting, might return either value depending on completion timing. The caching layer executes operations in the exact order they were initiated, regardless of completion order.

These features eliminate subtle bugs that are difficult to reproduce in testing, reducing the need for database expertise to write correct code.

Optional bypass

Gates and caching benefit most use cases, but not all. Some applications can tolerate concurrency safely, some prefer minimizing latency over write confirmation, and some access patterns make caching wasteful. For these, explicit bypass flags exist:

this.storage.get("foo", {allowConcurrency: true, noCache: true});
this.storage.put("foo", "bar", {allowUnconfirmed: true, noCache: true});

Developers who have reasoned carefully about these trade-offs can tune performance. Those who prefer not to think about it get sensible defaults.

The value of correctness at scale

Concurrency is hard at every experience level — even experts regularly get it wrong, because it's difficult to reason about all the ways operations can overlap and corrupt state.

The traditional answer — stateless applications with database transactions — is slow, contributing to the hundreds of milliseconds many web apps take for basic actions. Durable Objects embrace state, keeping it in memory alongside disk storage and routing requests for the same data through the same instance for speed. But that speed previously came with significant correctness complexity.

With input gates, output gates, and caching, intuitive code works correctly and runs fast. Developers can build applications without spending time optimizing I/O performance and debugging obscure race conditions.