Your Circuit Breaker is Misconfigured - Shopify

When Circuit Breakers Make Things Worse

Circuit breakers are a powerful resilience tool, but their effectiveness hinges entirely on configuration. A slightly misconfigured circuit can be as damaging as no circuit at all — changing one or two parameters can swing your system from running smoothly to complete failure. Understanding how your application behaves during service outages and knowing what each parameter actually controls is essential for predictable failure modes.

At Shopify, resilient fallbacks are integrated throughout the application stack. A fallback is a backup behavior that activates when a component or service goes down. For example, when Redis (which stores sessions) fails, users don't see an error. Instead, the problem is logged and pages render with sessions soft-disabled. This isn't just a matter of catching exceptions from a failing service — the real challenge is handling the period between when the service fails and when the fallback actually kicks in. Without protection, a Redis outage causes every connection attempt to time out, each lasting 2 seconds. Responses become unbearably slow, and request threads sit idle waiting for timeouts, driving utilization toward 100%.

During an outage, a worker that normally processes 5 requests per second drops to half a request per second — a tenfold throughput decrease. At that point, the service might as well be completely down.

Semian's Configuration Surface

Shopify's Semian circuit breaker implements the standard pattern: if a timeout is observed once or more, it's likely to keep happening until the service recovers. Rather than hitting the timeout repeatedly, the resource is marked dead, and calls to it raise instantly. An alternative library, Hystrix by Netflix, shares the core functionality but exposes fewer tuning parameters, which — as the details below show — can make it lose effectiveness for capacity preservation.

At first glance, Semian's configuration looks simple — just five parameters:

• name
• error_threshold
• error_timeout
• half_open_resource_timeout
• success_threshold

But treating these as arbitrary numbers or best guesses is a mistake. Each parameter interacts with the others and with the real-world request patterns your worker sees. The difference between a complete outage and a slight delay can be dramatic — a configuration change described later drops utilization during failure from 263% to 4%.

The following analysis uses the reference frame of a single worker, since circuit breaker state isn't shared across workers. All examples model instances of Redis — a common dependency at Shopify — but the logic applies to any service you protect.

Isolating Circuits Per Instance

The name identifies the resource being protected, and each name gets its own circuit breaker. Different service types (MySQL, Redis, etc.) must each have a unique name so that excessive timeouts in one service only open that circuit. But there's a subtler point: a single worker can talk to multiple instances of the same service type, sometimes dozens of Redis instances. An outage on one instance shouldn't kill all Redis connections, so each instance needs a distinct name, like redis_cache_#{instance_number}.

You need to know how many services your worker can reach, because every failing service adds to overall utilization. Define failing_services as the maximum number of simultaneous failures you need to handle — if you have 3 Redis instances but only care about the scenario where 2 fail, failing_services should be 2, not 3.

Error Threshold vs. Timeout Window

error_threshold counts how many errors must occur within an error_timeout window before the circuit opens. A larger threshold means the worker spends more time waiting on I/O before reaching the open state. Consider a single Redis instance failure with error_threshold = 3 and failing_services = 1: three consecutive timeouts happen, then the circuit opens and all further requests raise instantly.

With three failing Redis instances, the situation worsens. Each circuit requires three timeouts before opening, and all of them must open before the worker stops blocking on I/O. In a scenario with 40 Redis instances, each with a 1-second timeout and an error_threshold of 3, the minimum time to open all circuits is roughly 2 minutes. And this is optimistic — request order isn't guaranteed, so the actual time can be longer.

To keep the initial utilization spike low, minimize error_threshold. But awareness of false positives is critical: a lower threshold increases the chance that transient blips open the circuit incorrectly even when the service is healthy. At a steady-state timeout error rate of 0.1% in the error_timeout window, an error_threshold of 3 yields a false-positive probability of roughly 0.0000001%. That trade-off must be balanced against the fact that an erroneously opened circuit raises instantly for every request during the full error_timeout interval.

The Two Parameters That Matter Most

error_timeout serves dual duty: it defines both the measurement window for error_threshold and the duration the circuit stays open before retrying the resource. A larger value slows recovery after an outage and lengthens the negative impact of any false-positive circuit opening.

The critical design decisions emerge from the interaction between error_timeout and half_open_resource_timeout (detailed below). On the surface, you'd want to minimize error_timeout to shorten recovery time. But the analysis shows that maximizing error_timeout actually preserves worker utilization better in steady-state failure scenarios — the key insight is that a longer closed-circuit period combined with precise half-open probing can sustain throughput far more effectively than rapid retry cycles.

These two parameters form the core of a correctly configured circuit breaker. The remaining parameters — success_threshold and half_open_resource_timeout — govern what happens in the half-open state, when the circuit allows a limited number of test requests through to probe whether the service has recovered. Their settings determine how long that probe lasts and how many consecutive successes are required before the circuit fully closes again. A probe that's too aggressive (short timeout, low success_threshold) risks reopening the circuit immediately on the next hiccup; one that's too conservative (long timeout, high threshold) prolongs the degraded state after recovery.

The real-world consequence: a configuration with balanced error_timeout and half_open_resource_timeout keeps utilization stable during outages — you see a brief climb before the circuit opens, then a plateau where users experience only slight delays. The alternative, a mismatched pair of parameters, can drive utilization to 263% (a complete outage), whereas careful tuning holds it at 4% (a manageable, temporary degradation).

The Half-Open Penalty

When a circuit enters the half-open state, it allows a real request through to test whether the service has recovered. This happens after error_timeout has elapsed. If the service is still down, that probing request times out, the circuit opens again, and the cycle repeats. With one failing service, this is a minor annoyance. With many failing services, it becomes a steady drain on resources.

The periodic flip-flop between open and half-open states is deterministic, which means the wasted time can be calculated precisely. The obvious lever is error_timeout: increase it and you reduce how often the circuit wastes time on probe timeouts. But a larger error_timeout slows recovery and keeps the circuit open longer after false positives. With 40 Redis instances and a 1-second service timeout, that's 40 seconds of wasted timeout waits per cycle.

The other lever is the service timeout itself. Lower it and less time is wasted waiting. But the service timeout is often constrained by how long the underlying service legitimately needs to respond. This creates a tuning deadlock: you can't raise error_timeout without hurting recovery, and you can't lower the service timeout without breaking the service contract.

Semian's half_open_resource_timeout parameter addresses this directly. It's a separate timeout used only when the circuit is in the half-open state, replacing the original service timeout for those probe requests. This gives you an independent knob: a small half_open_resource_timeout relative to error_timeout minimizes wasted utilization without affecting normal service timeouts.

With three failing services, each circuit makes one timeout attempt in the half-open state before opening. All circuits must be open before the worker's I/O blocking stops, so three failing services mean three wasted timeouts per cycle.

Consider error_timeout = 5 seconds and half_open_resource_timeout = 1 second. A steady state cycle lasts 8 seconds total: 5 seconds of useful work and 3 seconds wasted waiting on I/O timeouts across the three services. That's 37% of utilization consumed by I/O waits.

Notably, Hystrix has no equivalent to half_open_resource_timeout, which can make it impossible to achieve a usable steady state for applications with many failing services.

Closing the Circuit on Success

The success_threshold controls how many consecutive successful probe requests are required before the circuit closes and accepts all traffic again. Its impact is most visible during partial outages where the error rate is below 100%.

With a 90% error rate and success_threshold of 1, the circuit will open and close frequently, since any single success closes it. There's a 10% chance per probe of closing the circuit when it shouldn't, and each close/reopen cycle adds I/O strain on the system.

Raising success_threshold to 3 changes the math significantly. Three consecutive successes are now required, and the probability of a spurious close drops to 0.1% per cycle. The tradeoff is that partial outages produce less flapping, but the system takes longer to recognize genuine recovery.

Hystrix also lacks an equivalent to success_threshold, making it hard to mitigate flip-flopping during partial outages.

Quantifying Utilization

Every circuit breaker parameter influences wasted utilization. The Circuit Breaker Equation models this behavior for the steady-state failure scenario, where the circuit continuously probes the half-open state. The equation does not account for thread context-switch overhead; applications with significant context-switch costs should use fewer threads.

Live testing validated the equation: observed utilization closely matched predictions.

A Tuning Example

Consider a Rails worker with 2 threads, integrating circuit breakers for 42 Redis instances. Each Redis instance has its own circuit with a service timeout of 0.25 seconds.

Starting parameters assume all 42 instances fail simultaneously — the worst case:

Plugging these numbers into the Circuit Breaker Equation yields an additional utilization requirement of 263%, which is unacceptably high. A target under 30% leaves room for normal traffic variation.

Production metrics show that 99% of Redis requests complete in under 50ms. With that headroom, half_open_resource_timeout can drop to 50ms while still allowing the circuit to close reliably when Redis recovers. Increasing error_timeout to 30 seconds slows recovery but dramatically reduces worst-case utilization.

With these adjustments, the additional utilization requirement drops to 4%.

The equation serves as a concrete reference for evaluating circuit breaker tuning decisions. Note that success_threshold does not affect steady-state utilization, since a single error is sufficient to keep the circuit open again once the probe fails.