Worker sharding: a new approach to an old cold start problem
Cloudflare first tackled cold starts in Workers five years ago by pre-warming Workers during the TLS handshake of their first request. The technique relied on the SNI sent in the very first message of the handshake, which gave enough information to prepare the target Worker before the request arrived. That approach worked well when Workers were small and fast to start. But the platform has changed since then: Workers are no longer constrained by the original 1 MB script size limit and 200 ms startup CPU budget. Both limits have been relaxed — script size grew to 5 MB, then 10 MB for paid users, and to 3 MB for free users; startup CPU time doubled to 400 ms. Meanwhile, TLS handshakes have not gotten slower. In fact, TLS 1.3 requires just one round trip, versus three for TLS 1.2, and is far more prevalent than it was when the original cold start work shipped.
Those changes meant that cold starts for complex applications could now take longer than the TLS handshake itself. The pre-warming approach still helps reduce visible delays, but it is no longer sufficient to hide them entirely. Cloudflare has now deployed a different mechanism, called Worker sharding, that reduces the absolute number of cold starts rather than trying to make each one faster.
What makes a cold start expensive
A Worker moves through a short lifecycle: it is instantiated from source code, serves requests, and is eventually evicted sometime after traffic stops. The eviction frees its resources for other Workers. The costliest phase is the initial instantiation and first invocation, which involves fetching the script source, compiling it, executing the module’s top level, and then running the initial request handler.
Routing requests to where Workers already exist
With relaxed limits, the time to perform a cold start grew in several ways: larger scripts mean more data transfer from storage, longer compilation, and potentially longer top-level execution, since the startup CPU budget increased. Rather than trying to shave time off each phase, the new approach focuses on reducing how often a cold start occurs at all — specifically, by routing requests to existing Worker instances instead of starting fresh ones.
Previously, Cloudflare could coalesce requests onto a single Worker instance only if they happened to arrive at a machine already hosting it. If a request landed on any other server in the data center, it would trigger a cold start on that machine, even if a warm instance existed elsewhere. This led to a counterintuitive situation: users prototyping with low traffic experienced high latency, while those with sufficient traffic saw their latency drop, simply because their Workers were less likely to be evicted between requests.
If requests are instead coalesced onto one server, the Worker receives traffic frequently enough to avoid eviction, producing a single cold start followed by a high warm request rate. Coalescing also stretches memory further — serving the same traffic from one server uses 299/300 less memory than spreading it across 300 servers. The memory savings reduce eviction pressure for other Workers too, creating a virtuous cycle.
The added latency from proxying requests to a different server internal to the data center is typically under one millisecond, which is far less than the cost of any cold start. The tradeoff is worth it in nearly every case.
A consistent hash ring for Workers
The solution mirrors a long-standing design in Cloudflare’s HTTP cache. When a request for a cacheable asset comes in, the routing pipeline chooses a cache server based on the request’s cache key — the URL plus other details. This “sharding” approach keeps the cache hit rate high by always sending the same asset to the same server. Each data center effectively contains one logical HTTP cache spread across all its servers.
Until now, the same could not be said for Workers. Each server held its own set of Workers, with no coordination between them. The new system borrows the cache’s data structure, a consistent hash ring, to give each Worker a stable home server within a data center. A naive mapping from Worker script IDs to server addresses would require full rehashing whenever servers come and go — which happens regularly due to crashes, maintenance, or decommissioning. A consistent hash ring maps both scripts and servers onto a number line that wraps around into a ring. To find a Worker’s home, its hash position is located on the ring, and the nearest server at or after that position is chosen.
When servers are added or removed, only the Workers whose hash positions fall before the affected server are re-homed. All other Workers stay put, avoiding a flurry of unnecessary cold starts.
Handling overload on shard servers
HTTP assets are static and their serving costs scale with size, which makes them easy to shard. Workers are different: they are live compute units that can use up to five minutes of CPU time per request. A single Worker instance can be overwhelmed if all of its traffic is directed to one server. The system therefore must always support horizontal scaling to handle traffic spikes, and a shard server must be able to refuse a request when it is overloaded. Shard clients — servers that initially receive requests and look up the shard server via the ring — must handle refusals gracefully without serving errors.
Two general approaches exist for shedding load without errors. The first requires the client to ask permission before sending the request, similar to Expect: 100-continue semantics in HTTP. This adds a round trip of latency just to establish that the request can be sent. The second approach sends the request immediately and relies on the server to forward it elsewhere — possibly back to the client — if it must refuse. This avoids the latency penalty but puts the shard server in the request path, potentially creating a trombone effect where request bytes bounce back and forth.
Sending sharded requests optimistically
Cloudflare chose the optimistic approach — send the request without first asking permission — for two reasons. First, refusals are expected to be rare. When a shard client receives a refusal, it cold starts the Worker locally and then serves future requests itself, making the shard server unnecessary for that Worker until the traffic drops enough to cause an eviction. Second, Cloudflare has a mechanism to avoid the trombone effect.
Cross-instance communication in the Workers runtime uses Cap’n Proto RPC. When assemblying a sharded request, the shard client includes a capability — Cap’n Proto’s term for a distributed object handle — to a lazily-loaded local instance of the Worker. This instance exposes the same interface as any other Worker, but it gets cold started only when actually invoked. If the shard server must refuse the request, it does not send back a “go away” response. Instead, it returns the shard client’s own lazy capability.
The shard client’s application code sees only that it received a capability; it does not know where that capability is implemented. But the RPC system recognizes the capability as local and knows that any bytes sent onward will loop back. It stops sending request bytes, waits to receive back what it already sent, and shortens the request path as fast as possible, taking the shard server out of the loop entirely.
Sharding nested Worker invocations
Workers routinely invoke other Workers, whether through Service Bindings or features like Workers KV. The most demanding case is Workers for Platforms, which involves three types of Workers: a dynamic dispatch Worker, any number of user Workers, and an optional outbound Worker. A typical flow starts with the dynamic dispatch Worker, which selects and invokes a user Worker. That user Worker may then invoke the outbound Worker, which intercepts its subrequests. The dynamic dispatch Worker may also have a tail Worker, which must receive trace events for the entire request flow in a single invocation, even if the flow fans out across multiple machines.
Supporting this nesting requires a context stack that tracks ownership overrides, resource limit overrides, trust levels, tail Worker’s configurations, outbound Worker’s configurations, feature flags, and more. When everything ran on a single thread, moving this stack around was manageable. To shard a Worker deep inside an invocation chain, the stack must serialize into a Cap’n Proto data structure and travel with the request to the shard server, where it is deserialized back into native objects.
The distributed object model resolves otherwise difficult problems. To coalesce tracing data from invocations that were fanned out across any number of servers, the dynamic dispatch Worker’s home server creates a capability for a reportTraces() callback and embeds it in the serialized context stack. Every shard server that receives the context stack can call that callback without knowing where the home server actually lives. The same pattern applies to each sharded Worker in the stack, no matter how deep the nesting goes.
Results: fewer evictions, higher warm request rates
The rollout produced immediate efficiency gains. Only about 4% of total enterprise requests ended up being sharded. That is, 96% of enterprise traffic targets Workers that are loaded enough to require multiple instances per data center. The sharded requests are all to low-traffic Workers, which benefit most from co-location.
Despite the low sharding rate, the global Worker eviction rate dropped by 10x. Eviction rate is a measure of memory pressure, akin to garbage collection at a macro level. Fewer evictions mean memory is used more efficiently, so Workers are kept in memory an order of magnitude longer and their warm request rate improves. This high leverage comes from the power-law distribution of Internet traffic: a small number of Workers receive a huge share of requests, while a long tail of Workers receives very few.
For enterprise traffic, the warm request rate rose from 99.9% to 99.99% — from three nines to four nines. Equivalently, the cold start rate fell from 0.1% to 0.01% of requests, a 10x decrease, which is consistent with the 10x reduction in evictions. Warm request rates also became more stable throughout the day, though there remains some room at the top of the curve before reaching five nines.
Cold starts have not been completely eliminated. But this approach reduces how often they occur at all, rather than merely attempting to make each one faster.
