Slowing Down to Speed Up – Circuit Breakers for Slack’s CI/CD

Orchestration-Level Circuit Breakers: Managing Slack’s CI/CD Load

By 2020, Slack’s internal CI/CD tooling had reached a breaking point. The company’s developer headcount and the complexity of its test suites were both growing at roughly 10% month-over-month. Internal services—many of them interdependent—began failing under the load, triggering cascading failures that left engineers without reliable builds, tests, or deployments for days at a time.

The short-term fixes were familiar: scale the GitHub Enterprise appliance to the largest available AWS instance, and add nodes to whichever service happened to be the bottleneck. But each fix only moved the problem to the next service in the chain. Slack needed a way to regulate the flow of requests between internal services rather than simply scaling each one to absorb the new peak.

The Interlocking Scale and Complexity Problems

Slack’s architecture had grown well beyond the original single Webapp service. Clients connect to multiple APIs, and the engineering organization had grown to match that complexity. Automated tests multiplied alongside features, and the CI/CD system—called Checkpoint—orchestrated the resulting builds, tests, and deployments through a set of tightly coupled services.

Several failure patterns emerged:

• Git appliance saturation: When peak daily test counts exceeded what the Git application could serve, engineers encountered Git errors. This caused Checkpoint to enqueue more retry jobs, which created backlogs in both Checkpoint’s job queue and Jenkins. Engineers continued working while tests were unavailable, generating even more jobs before the initial queue could drain.
• Queue-based positive feedback: Checkpoint’s internal asynchronous job queue sits on self-hosted main-main MySQL (later migrated to AWS RDS Aurora). A scheduler limits concurrent jobs to protect the database, but when the queue grows too long, users re-request the same jobs—creating a loop that grows the queue further.
• Hidden downstream limits: Slack’s tooling engineers had repeatedly scaled test executors and environments to support growth. The cumulative effect was that the CI search cluster became the bottleneck, and requests from test executors and test environments began failing, adding even more load.

Scaling individual services was not sustainable. Some services, like the Git appliance, had already reached the limits of vertical scaling in AWS. Others, like the search cluster, could not absorb new request types without failing under the additional concurrency. Slack needed to treat the problem not as a path to scale but as a question of how to handle load when downstream systems were saturated.

Circuit Breakers at the Orchestration Layer

Slack engineers—borrowing a concept from systems engineering and Michael Nygard’s work on fault tolerance—implemented circuit breakers to detect faults in external systems and stop sending calls to those systems when they were overloaded or erroring.

The key architectural decision was where to put them. Rather than implementing circuit breakers in each individual client service, the team placed them in the Checkpoint orchestrator's background job scheduler. Checkpoint sits at the junction between multiple internal systems: it receives webhooks from GitHub, enqueues background jobs to trigger builds and tests in Jenkins, and updates state in its database. By controlling the flow of requests at this orchestration layer, a single circuit breaker implementation could regulate the interface between many pairs of systems.

The circuit breakers had three states: closed (normal operation), open (requests are deferred or dropped), and, notably, no manual half-open state. In traditional circuit breakers, a half-open state allows a small trickle of requests through to test recovery. Checkpoint’s background job system already provides retries, and the circuit breaker includes a TTL on the Prometheus query. When the TTL expires, the circuit checks again and resumes if the dependent service has recovered.

The Checkpoint team built the initial abstraction in Hacklang (Slack’s variant of PHP). The implementation directly pulled health metrics from multiple dependent services using Prometheus queries via Trickster, a proxy and cache that fronts multiple Prometheus clusters.

namespace Checkpoint\CircuitBreaker;
use type Slack\Checkpoint\PromClient;

/*
* Generic interface for Circuit Breakers in Checkpoint.
* Downstream actions include deferral mechanisms or load shedding.
* @see https://martinfowler.com/bliki/CircuitBreaker.html
*/

enum CircuitBreakerState: string {
  CLOSED = 'closed';
  OPEN = 'open';
}

abstract class CircuitBreaker {

  /**
   * Get the state of this circuit breaker. Note the return value is intentionally
   * not a `Result<CircuitBreakerState>`. In the case of internal errors, this must
   * decide if the breaker fails open/closed.
   */
  abstract protected function getState(): CircuitBreakerState;

  /**
   * Allow for bypassing a circuit breaker. Used as a circuit breaker for circuit breakers.
   * In a subsequent class, add the following to always allow the request to pass through
   * <<__Override, __Memoize>>
   * public function bypass(): bool { return true; }
   */
  public function bypass(): bool {
    return false;
  }

  public function allowRequest(): bool {
    $state = $this->getState();
    PromClient::circuit_breaker_requests()->inc(1, darray[
      'breaker_type' => (string)static::class,
      'breaker_state' => (string)$state,
    ]);
    if ($this->bypass()) return true;
    return $state === CircuitBreakerState::CLOSED;
  }
}

When a circuit was open, Checkpoint deferred or dropped the request. This served two purposes: it reduced the load on the failing service, and it removed the queue buildup that had previously created cascading failures.

Implementation and Service Coverage

The first sprint targeted orchestration service health directly:

• Test job deferral: When Checkpoint’s own job queue or Jenkins queue passed a threshold, test jobs were deferred rather than executed.
• Environment saturation deferral: End-to-end test jobs were deferred when all Slack test environments were busy.
• Load shedding for stale commits: Test executions for older commits on a branch were dropped to prioritize the latest code.
• Retry load sheduffing: Test retries for suites that had experienced consistent failures were dropped to avoid hammering a broken system.

A second sprint extended circuit breakers to shared dependent services:

• Flannel: an application-level edge cache serving frequently fetched teamwide data across multiple regions.
• Vitess: the database platform that provides the source of truth for all customer data, queried with MySQL syntax.
• Search: the service that computes indices of messages, files, and people, maintaining both a real-time live collection and a weekly offline collection.

The implementation for each service followed a similar pattern, as shown in the simplified Flannel example. Each circuit breaker performs a cache lookup (with a TTL), runs a Prometheus range query via Trickster, and sends user-facing messages about deferrals or load shedding. If the metric query itself fails—for example, if Trickster or Prometheus returns an error—the circuit breaker remains closed to avoid blocking traffic on a monitoring problem. Consistent responses are cached between asynchronous jobs to reduce query load.

namespace Checkpoint\CIBot\CircuitBreaker;

use namespace Checkpoint\{CIIssue, Trickster};

use type Checkpoint\CIBot\Delta\{DeltaAnomalyType, DeltaDimensionType};
use type Checkpoint\CIIssue\ServiceDepCircuitBreakerType;
use type Checkpoint\CircuitBreaker\{Cacheable, CircuitBreaker, CircuitBreakerState};
use type Slack\Checkpoint\PromClient;

type flannel_callback_error_rate_cache_t = shape(
	'ts' => int,
	'error_rate' => int,
);

final class FlannelServiceDepCircuitBreaker extends CircuitBreaker {
   use Cacheable;

   const int TTL = 60; // Time-to-Live for cached value
   const int FLANNEL_CALLBACK_ERROR_RATE_THRESHOLD = 5;
   const string PROM_FLANNEL_CLUSTER = 'flannel';
   const string PROM_FLANNEL_QUERY_GLOBAL = 'sum(dc:cb_errors:irate1m{error!~"org_login_required"})';
   const string ISSUE_MESSAGE_OPEN = ':warning: :broken_yellow_heart:  Flannel Circuit Breaker is open. Tests are deferred';
   const string ISSUE_MESSAGE_CLOSE = 'This circuit breaker is closed. Tests are starting again';
   const string ISSUE_KEY = ServiceDepCircuitBreakerType::FLANNEL;

   public function __construct(private ?\github_repos_t $repo = null, private ?\TSlackjsonValidatorPropertiesCheckpointPropertiesTestsItems $test = null) {}

   <<__Override, __Memoize>>
   public function getState(): CircuitBreakerState {

       $cached_key = $this->getCacheKey(self::class, 'flannel_callback_errors');
       $cached_data = \cache_get($cached_key);
       $existing_error_rate = 0;

       // If the cache exists, and is fresh enough, use it. Default to Closed
       $result = \type_assert_type($cached_data, flannel_callback_error_rate_cache_t::class);
       if ($result->is_error()) { return CircuitBreakerState::CLOSED; }

       $data = $result->get();
       $existing_error_rate = $data['error_rate'];
       if ($this->isValidCache($data['ts'], static::TTL)) {
           if ($existing_error_rate < static::FLANNEL_CALLBACK_ERROR_RATE_THRESHOLD) {
               return CircuitBreakerState::CLOSED;
           } else {
               return CircuitBreakerState::OPEN;
           }
       }
       // Lets fetch the current error rate (and compare against the former one)
       $result = $this->getFlannelCallbackErrorRate();
       if ($result->is_error()) {
           return CircuitBreakerState::CLOSED;
       }

       $error_rate = $result->get();
       $cached_value = shape('ts' => \time(), 'error_rate' => $error_rate);
       \cache_set($cached_key, $cached_value);

       if ($error_rate >= static::FLANNEL_CALLBACK_ERROR_RATE_THRESHOLD) {
           PromClient::cibot_service_dependency_error_rate_above_threshold()->inc(1, darray[
               'breaker_type' => (string)static::class,
           ]);

           if ($existing_error_rate < static::FLANNEL_CALLBACK_ERROR_RATE_THRESHOLD) {
               CIIssue\send(static::ISSUE_MESSAGE_OPEN, DeltaDimensionType::CIRCUIT_BREAKER, DeltaAnomalyType::CIRCUIT_BREAKER_OPEN, static::ISSUE_KEY);
           }

           return CircuitBreakerState::OPEN;
       }

       // If our circuit breaker was previously open (and now closed), track this new state and mark it in our issues dataset
       if ($existing_error_rate >= static::FLANNEL_CALLBACK_ERROR_RATE_THRESHOLD) {
           CIIssue\end(static::ISSUE_MESSAGE_CLOSE, DeltaDimensionType::CIRCUIT_BREAKER, DeltaAnomalyType::CIRCUIT_BREAKER_OPEN, static::ISSUE_KEY);
       }
       return CircuitBreakerState::CLOSED;
   }

This approach shifted the incident response model. Instead of discovering a service was failing after it had already affected developers—and then spending multiple days assembling teams to restore service—Checkpoint could detect the strain on a downstream system in near real time and apply coordinated deferral. Developers saw a slower CI/CD pipeline during periods of high load, but the system avoided the customer-visible cascading failures of the past and preserved availability for the most important workflows.

How Slack surfaces CI/CD failures

Open circuit breakers automatically post to Slack channels, complete with links that show different views of the same underlying issue. A typical flow starts with an engineer noticing the alert, then escalating to the appropriate team channel for follow-up.

Similar deferred-message notifications appear in the Checkpoint frontend, so engineers see immediately when their tests are held back due to a service problem.

To track these issues over time, Slack built a small internal library that reports open circuit breakers to Slack. The same library was later extended to handle anomaly detection—flagging higher-than-expected failure, error rate, duration, or flakiness across test executors, environments, and suites.

Impact on developer experience

The two sets of circuit breakers—one for internal infrastructure, one for dependent services—have reduced the surface area for cascading failures by deferring tests, and smoothed test throughput by shedding load. Slack reports zero cascading failure incidents in internal tooling over the last two years, a category of incident that was common before the breakers were introduced in 2020.

The improved feedback loops are a key benefit. Engineers now learn from Checkpoint's frontend and Slack when their tests are deferred until the system recovers. Previously, those tests would flake or fail due to an overloaded downstream system. Deferring tests has meant fewer flakes and fewer test executions that are no longer relevant to the engineer who submitted them.

As a side effect, Slack was able to recalibrate spend and provisioning for the portion of its fleet dedicated to testing stale commit SHAs, contributing to a cumulative 10x decrease in spend versus baseline growth.

One metric that teams watched closely was “time to test results,” the lag between a build and test results reaching the developer. There was concern that deferring or shedding load would slow this metric. Instead, it has stayed steady, since many deferred tests would have failed and produced flaky results anyway.

Adapting the pattern to other organizations

CI/CD culture, tooling, and infrastructure differ widely across companies, so this solution can't be copied verbatim. The useful part to borrow is the concept of orchestration-layer circuit breakers that separate concerns.

Slack focused on Webapp, the main repository housing business logic and the majority of developer attention. Its test environment is complex: many Webapp tests depend on both synchronous and asynchronous requests, such as an API call that enqueues background jobs for notifications or search. Slack aligned its testing taxonomy with Google's small/medium/large model, pushing toward more medium (integration) tests to verify safety rather than relying mostly on unit and end-to-end tests.

For organizations considering this approach, Slack recommends using metrics, events, logs, or traces to identify common pipeline problems. Its own rollout started by sharing awareness of open circuit breakers in Slack before actually deferring or shedding requests, building confidence iteratively.

In Slack's case, the complex service layout of Webapp drove a corresponding test structure. The company is now moving toward continuous deployment and automated rollbacks later in the pipeline.

Bottom line

Orchestration-level circuit breakers in Checkpoint have eliminated cascading failures between internal tooling systems since the project landed in 2020. Engineers see fewer flakes, better service availability, and smoother throughput from Checkpoint, with a measurable impact on productivity. Multiple teams are now applying the same pattern of programmatic metric queries to move Slack toward automating builds, tests, deploys, releases, and rollbacks.