CI at Slack: A Case Study in Data-Driven Infrastructure Change
Slack’s Core Development Engineering department supports the engineers who build, test, and release the company’s services. A central focus is the stability and throughput of the continuous integration (CI) systems that underpin this work. Slack’s CI evolved alongside its architecture, which grew from a PHP monorepo (now mostly Hack Lang) into a topology of many services. For webapp — where most engineers still spend their time — the CI ecosystem is built around Checkpoint, an internal orchestration service that manages complex workflows and a job queue, and Jenkins executors that run builds and tests. A companion tool called cibot executes the build and test scripts, communicating back to Checkpoint for orchestration and result reporting.
From Growth to Curve Changes
Slack’s rapid expansion in customers and codebases introduced complexity, blurred service boundaries, and pushed internal tools to their limits. From April 2017 (when the first test results were recorded) to April 2020, the company saw an average of 10% month-over-month growth in test suite executions.
To manage this growth, the team relies on detailed analytics dashboards tracking metrics like execution counts, errors, flakiness, and timeouts. These dashboards inform hypotheses and project scoping, helping the team assess the impact of large-scale changes. The team categorizes its infrastructure optimization work into three types of "curve changes":
- Adaptive capacity — decreasing the cost per test by changing the infrastructure runtime.
- Circuit breakers — decreasing the total number of tests by changing the infrastructure workflow.
- Pipeline changes — decreasing the number of tests by changing user workflows.
Adaptive Capacity: A Case Study in Parallelization
One project began with a request from the cloud infrastructure team to move off older-generation AWS compute-optimized instances, promising a 10% cost saving and better performance. Initial scoping dashboards revealed a heterogeneous fleet of instance types and generations, along with extremely spiky workloads — daily at 1 pm Pacific, a deluge of requests would sometimes overwhelm the infrastructure.
Rather than attempting to find a single "right-sized" instance, the team recognized that ideal instance types are often a myth, given the diverse and changing nature of CI workloads. They established three guiding tenets:
- Increase peak fleet capacity to handle spiky workloads.
- Standardize on one instance type to reduce maintenance overhead.
- Measure user experience with end-to-end metrics for performance, resiliency, and cost-per-test.
The strategy centered on adaptive capacity and oversubscription. Previously, most CI executors ran on dedicated instances with no parallelization. The hypothesis was that, for non-compute-heavy tasks (like network calls), CPU—the key bottleneck—could be shared. By increasing parallel executions per instance (to 2 or 4 executors), the team aimed to increase total throughput without degrading performance or raising error rates.
Profiling across four representative build and test suites validated this approach. Some test suite owners had to refactor workflows that had assumed a single executor per instance. The team built a model based on end-to-end profiling across instance types with varying vCPUs and executor counts, aiming for a profile that balanced cost and performance while keeping error rates at zero. For instance, a c5.9xlarge instance with 4 executors showed no additional faults and only a 5% performance degradation versus 2 executors. The team now targets 50% busy executors at peak to maintain a buffer for sudden demand.
Cutting Off Failure at the Source
Slack's CI orchestration platform now uses circuit breakers on downstream services and user workflows. Borrowed from systems engineering, these breakers detect faults in external systems and stop sending calls to the known-faulty component, periodically probing for recovery so CI can resume normally. In practice, this means CI jobs can be deferred or cancelled entirely—for instance, when a user pushes a new commit to a branch, making previously queued jobs obsolete.
Before this work, cascading failures from webapp issues routinely took down the CI pipeline, resulting in multi-day severity 2 incidents (one step below a full outage) that frustrated developers across Slack and forced internal tools teams into scramble mode. Engineers and test suite owners had no visibility into the performance or resiliency of the underlying test infrastructure, leading to misplaced blame and delayed fixes.
The team implemented circuit breakers for internal CI services—Jenkins, GitHub commits, and Checkpoint's job queue—as well as for the services CI depends on, including Flannel, Vitess, and Search. Breaker state is surfaced in team alert channels and in the CI frontend, letting users and operators know when a downstream system is degraded.
Observability-Driven Pipeline Changes
By mid-2020, flaky test executions were the top reported developer productivity issue for multiple quarters and represented significant CI infrastructure spend. An earlier push to add tracing to CI code had already improved distributed systems understanding, but pipeline changes required more: the team worked directly with CI customers to examine test executions under trace instrumentation, aiming to cut both developer wait time and infrastructure spend on faulty code.
At peak flakiness in late 2020, Slack's Automation teams held daily 30-minute triage sessions on the flakiest test suites. Team leads were hesitant to change how they used the Cypress end-to-end test framework, believing the flakiness originated in the test code itself—yet focusing on individual tests and suites yielded little progress.
The team agreed on a different hypothesis: instrument the Cypress platform runtime with traces for a month to capture runtime variables, after verifying there was no measurable performance or resiliency cost. Within a week, several runtime variables showed a strong correlation with higher flake rates. At peak, Slack was burning roughly 90,000 compute hours per week on very large, very expensive machines for tests that were ultimately discarded as flaky. To build confidence at every merge and hypothesis test, the team queued up a revert PR alongside each change—but never needed it.
These tracing-driven workflow changes increased consistency and reduced flake rates across test executions. The resulting circuit breakers stabilize system throughput and give engineers a better experience: fewer flaky CI failures and no need to manually re-run failed tests.
What Changed
Slack drove a magnitude change in CI infrastructure spend through three connected ideas:
- Adaptive capacity to decrease the cost of each test by changing the infrastructure runtime.
- Circuit breakers to decrease the number of tests by changing the infrastructure workflow.
- Pipeline changes to decrease the number of tests by changing user workflows.
The work spanned multiple teams and required a shared curiosity about what the data actually showed. The result was not just reduced spend, but internal tools that behave more predictably and engineers who can trust their CI results.
