Modeling Queueing Behavior Before You Deploy It
Web applications behind a single-threaded server behave in predictable but often undesirable ways. A Rails process that can handle only one request at a time will queue everything else, and tail latencies balloon when a long request lands ahead of shorter ones. The math is straightforward: if the mean processing time is 100 ms and parsing adds 20 ms, a request that falls behind a slow peer waits for the entire queue to drain before it gets a turn.
This is exactly the scenario that motivated Timelike, a simulator from Aphyr that models network components as composable functions. Each component—cables, queues, load balancers, servers—accepts a request, advances a virtual clock, and passes the response back down the pipe. The result is executable infrastructure: you can test routing strategies and queue dynamics without standing up a single EC2 instance.
Simulated Time, Real Parallelism
Timelike doesn’t actually sleep. Instead, it virtualizes time across all threads in the simulation. Operations complete instantaneously relative to the virtual clock, and the clock advances only when a thread explicitly sleeps. Because the simulation is deterministic with respect to wall-clock time, you can run the same model at 100 nodes or 1,000 and get comparable results.
The system also handles one of the subtler parts of concurrent simulation: deadlock. If a thread blocks on a mutex held by a thread that won’t release it until the clock moves forward, the scheduler detects the condition and advances time. It’s not the fastest path, but it keeps the model honest.
Random Routing: Simple but Uneven
The most basic load-balancing strategy is random selection. Pick a dyno for each incoming request, uniformly at random. Over an infinite stream of requests, this distributes load evenly. But infinite streams don’t exist. In practice, a random balancer occasionally stacks many requests onto one dyno while a neighbor idles. Every request in that dyno’s queue pays for the bad luck of the first one.
Random load balancing has one structural advantage: it requires almost no coordination. Balancers don’t need to share state or agree on routing decisions. They also compose cleanly. Two layers of random balancers behave identically to one. If a fleet of independent routers must act without central knowledge, random is often the only feasible baseline.
Round-Robin: Even Distribution, Not Even Durations
A round-robin balancer keeps a ring of backends and sends each request to the next entry in the list. Distribution becomes perfectly even by request count, and the state is trivial: one integer index into the ring.
The problem is service time. Real requests are variable. When two long-running requests land on the same dyno via round-robin, its queue depth spikes even though the pool as a whole has capacity. Round-robin protects against burst arrival patterns but not against burst durations.
Least-Connections: The Queue-Aware Option
Least-connections routing tracks how many active connections each backend is handling. A new request goes to the backend with the smallest current count. For a pool of single-threaded servers, that also means the backend with the shortest queue. This strategy cut 95th-percentile latency roughly in half compared to random routing, and median times by about a third in the simulator’s dyno model.
The ideal algorithm would predict request duration before it starts and place long jobs on idle nodes. That kind of foresight is impossible, so least-connections is the best queue-aware rule available with instant knowledge. It comes at a price: every balancer in a cluster must track connection counts across all backends, and that state must stay synchronized. In a tightly coupled cluster with a reliable network and moderate coordination cost, that’s a good trade.
Where Distribution Breaks Coordination
Heroku’s Bamboo architecture can’t run a single coordinating load balancer at its scale. The backends are spread too far apart for fast, reliable state sharing. Instead, Bamboo runs several independent least-connections balancers in front of the dyno pool, each one maintaining its own view of connection counts.
The problem with this arrangement is that two routers can dispatch requests to the same dyno simultaneously. Neither sees the other’s connection count bump, and after the fact the dyno’s queue is deeper than either router predicted. Adding more least-connections routers in parallel degrades tail latency linearly, asymptotically approaching the behavior of random routing as coordination becomes thinner.
This is precisely the pattern visible in Heroku’s client-visible latency degradation. As the Bamboo cluster scaled, dyno queue-depth variability increased not because any individual router failed, but because the aggregate of weakly coordinated routers statistically recreated the same burst behavior that random routing exhibits. The simulator reproduces that dynamic in miniature, and it serves as a useful reminder: routing strategies that perform well in a single-balancer setup don’t automatically survive distribution. You can model that gap before you build it.
