When one Redis node isn't enough
Redis occupies a particular niche in production stacks: it's not the system of record, but it's the fast, in-memory layer for ephemeral data like metrics, sessions, and caches. Its single-node performance is remarkable—various sources cite throughput in the range of a million operations per second—but there are workloads that can saturate even that.
Stripe's rate limiters ran on exactly such a hot path. All rate limiting checks lived on a single Redis instance, with followers standing by only for failover. Every API request traversed multiple rate limiters, each requiring several Redis commands. The result was a node handling tens to hundreds of thousands of operations per second, with one core pegged at 100%.
Operating at maximum capacity, Redis degrades gracefully—mostly. The visible symptom was an elevated baseline of connectivity errors from clients configured with aggressive timeouts (~0.1 seconds) who couldn't get a connection or command executed in time. Most services tolerated this. The real trouble emerged when legitimate users authenticated successfully and launched high-concurrency scrapers, generating traffic orders of magnitude over allowed limits. Rate limiters defaulted to allowing requests under error conditions, which pushed that surge through to backend databases—systems that degrade far less gracefully than Redis under load.
Sharding without the coordination tax
Redis Cluster's design preserves the core value of speed. Unlike many distributed systems, it doesn't coordinate across nodes for each operation. Instead, it partitions the keyspace across independent Redis nodes, trading high availability guarantees for minimal overhead. Running an operation against a cluster costs negligibly more than against a standalone instance.
The keyspace is divided into 16,384 slots, and a client-side hashing function deterministically maps any key to one of them:
HASH_SLOT = CRC16(key) mod 16384
For a command like GET foo, the client computes the slot for the key foo:
HASH_SLOT = CRC16("foo") mod 16384 = 12182
Each cluster node owns a slice of these slots. Node-to-node communication handles slot distribution, availability, and rebalancing.
Clients discover the slot-to-node mapping through the CLUSTER command family. CLUSTER NODES returns the current topology, which clients cache locally and refresh as needed.
127.0.0.1:30002 master - 0 1426238316232 2 connected 5461-10922
127.0.0.1:30003 master - 0 1426238318243 3 connected 10923-16383
127.0.0.1:30001 myself,master - 0 0 1 connected 0-5460
The essential information in that output is the host addresses in the first column and the slot ranges in the last. A range like 5461-10922 means the node serves every slot between those two endpoints.
Redirecting with MOVED
When a node receives a command for a slot it doesn't own, it doesn't proxy. It replies with a MOVED error that names the correct node:
GET foo
-MOVED 3999 127.0.0.1:6381
MOVED is particularly important during rebalancing, when slots migrate between nodes and locally cached mappings go stale.
The cluster could theoretically have a node fetch the result from the correct owner and forward it, but MOVED is a deliberate design choice. It pushes some complexity onto clients in exchange for deterministic, single-hop execution as long as mappings are current. Rebalancing is rare, so the coordination cost amortizes to near nothing over the cluster's lifetime. (There are other cluster-specific mechanics at work, but the full specification covers those in depth.)
Cluster-aware clients need two capabilities beyond a standard Redis client: the key hashing algorithm and a maintained slot-to-node mapping. The typical client loop looks like this:
- On startup, connect to any node and fetch the mapping table with
CLUSTER NODES. - Execute commands normally, routing each key to the node that owns its slot.
- On
MOVED, refresh the mapping and retry.
Multi-threaded clients can optimize this by marking the mapping table dirty on MOVED, following the redirect for the current command, and letting a background thread refresh mappings asynchronously. During a rebalance, most slots don't move, so most commands continue without interruption.
Hash tags keep multi-key operations local
Rate limiting at Stripe relies heavily on EVAL with Lua scripts. A single EVAL is atomic, which is essential for correctly computing remaining quotas under concurrency. But a distributed keyspace breaks that model: keys that logically belong together—say user123.first_name and user123.last_name—could hash to slots on different nodes, making a cross-key EVAL impossible without expensive remote fetches.
Consider a script that concatenates those two keys to build a full name:
# Gets the full name of a user
EVAL "return redis.call('GET', KEYS[1]) .. ' ' .. redis.call('GET', KEYS[2])"
2 "user123.first_name" "user123.last_name"
> SET "user123.first_name" William
> SET "user123.last_name" Adama
> EVAL "..." 2 "user123.first_name" "user123.last_name"
"William Adama"
Redis Cluster resolves this with hash tags. The rule: if a key contains curly braces, only the text inside them is hashed. Redis Cluster refuses multi-key operations whose keys don't share a slot—again optimizing for speed—so it's the application's job to guarantee locality. Rewriting the keys above to tag the shared identifier does it:
> EVAL "..." 2 "{user123}.first_name" "{user123}.last_name"
HASH_SLOT = CRC16("{user123}.first_name") mod 16384
= CRC16("user123") mod 16384
= 13438
With {user123}.first_name and {user123}.last_name, both keys map to the same slot, and the EVAL runs on a single node without issue. The same principle extends cleanly to a full rate limiter implementation, where multiple counters and timestamps for a single user or API key must be updated atomically.
Reliability via partitioning
Stripe's migration to a 10-node Redis Cluster had negligible performance impact. The more important benefit was operational: the failure cliff flattened because no single node was running at saturation. Horizontal scaling became a matter of adding nodes and letting the cluster redistribute slots. The rate limiting stack that once depended on one very hot instance now has a straightforward path to more capacity—and the ambient error rate that accompanied running at the edge of a single node's capability is gone.
Runway for growth
The migration to Redis Cluster proved far less disruptive than expected. The hardest part was hardening one of the cluster clients for production use, and even today client support can be uneven. That may simply reflect that most deployments stay on a standalone instance because a single Redis node is fast enough for their needs. Once we moved over, error rates dropped sharply, and we are confident the new architecture leaves ample room for continued growth.
Redis Cluster’s design has a philosophical appeal: simple, yet powerful. Distributed systems frequently become overly complicated, and that complexity can be catastrophic when a tricky edge case surfaces in production. Redis Cluster scales while keeping moving parts few enough that even a non-specialist can reason about its behavior. Its design doc is comprehensive but approachable.
In the months since setup, it has not required a single touch despite carrying considerable load every second of the day. That is a rare quality in production infrastructure, and one not even found among some other long-standing favorites like Postgres. The ecosystem needs more building blocks akin to Redis — components that do what they are supposed to do, then stay out of the way.
1 The exact number of operations per second is left intentionally vague.
2 Notably, we're not error-free. There are enough operations in flight that some level of intermittent failure is unavoidable.
