Slack Rebalances CI/CD Testing Without Losing Safety
For years, Slack's Webapp monorepo ran nearly all tests before code merged, with most of those tests being end-to-end suites. By 2020, the build-up had become painful: median PRs triggered more than 40 end-to-end suites (p95 over 60), test turnaround time at p95 regularly exceeded 30 minutes, and roughly 50% of PRs hit at least one flaky test. Even when individual suites ran reliably, sheer volume made flakiness a near-certainty across a full PR.
Slack's Core Development Engineering group responded with a tiered testing workflow that moved most test execution out of the pre-merge path and into new post-merge and regression pipelines. After roughly a year of production use, the results were significant:
- Test turnaround time (p95) dropped over 40%, staying consistently below 18 minutes.
- Test flakiness per PR fell over 90%, landing consistently under 5%.
- No measurable increase in customer-reported defects occurred.
The Problem With Pre-Merge Everything
Webapp combines a Hacklang backend and API with a TypeScript frontend. Slack introduced automated testing for the repo in 2017, and test execution grew roughly 10% month-over-month for the next three years. At the 2020 peak, the CI system ran one million test suites per day, with up to 40,000 tests per suite. Almost all of that ran before merge, and most were end-to-end tests exercising complex, interdependent workflows.
That created compounding problems. End-to-end suites failed for reasons unrelated to code changes: flaky tests themselves, test framework timeouts, unavailable services in the CI stack, and infrastructure issues like DNS outages. Engineers had to debug failures with no clear escalation path, and test owners often went unaware until something broke completely. The sheer volume also stressed internal CI infrastructure into cascading failures throughout 2019 and 2020.
Individual suite flake rates looked acceptable in isolation, but multiplied across dozens of suites per PR, the experience degraded sharply. With 60 suites at a 1% flake rate each, 55% of PRs would still see at least one flaky suite—an unsatisfying outcome despite "good" per-suite reliability. Keeping every suite pre-merge was unsustainable for engineering productivity, triage capacity, and infrastructure load. The question became how to cut the pre-merge burden without letting bugs through to customers.
A Three-Tier Pipeline Instead of One Big Gate
The solution replaced a single pre-merge wall with three testing tiers. The existing pre-merge pipeline still runs <1% of end-to-end tests, all for high-criticality features that block merges and deploys. Two new post-merge pipelines handle the long tail:
- Post-merge pipeline (new): A subset (<10%) of end-to-end tests runs against each commit merged to mainline. These cover medium-criticality features and block the CD pipeline before deploy.
- Regression pipeline (new): The remaining end-to-end tests batch-execute against batches of commits on a bi-hourly cadence. These cover low-criticality features and run post-deploy, reducing false negatives from flaky tests or platform hiccups.
The shift changes what a merge means. Pre-merge tests no longer gate every change; instead, a post-merge failure alerts the engineer and blocks the deploy path through CD. This flips the model from "prove everything before merge" to "catch and fix forward quickly," which required giving each test an explicit owner and escalation route.
Triage and Organizational Alignment
Moving tests out of pre-merge meant the team needed a dependable triage workflow to close the loop on failures. Slack created documentation and a self-driven course covering the new alert system, then had engineers shadow experienced triagers before rotating in. Escalation priorities are now clearly ranked: regression tests are lowest, followed by pre-merge, post-merge, and CD tests at the top.
High-priority tests were identified across teams and moved into the post-merge pipeline. These are held to a strict standard of under 0.1% flakiness and are meant to generate strong signals for important user experiences. Improved provenance tooling now maps each test to a business-area-specific channel, e.g. messaging tests route to #post-merge-messaging, with the owning team staffing a triage rotation. A shared #alerts-ci-issue channel raises awareness of decentralized CI issues, and bots automatically ping engineers on triage when failures occur. A cross-departmental working group of senior engineers from 12 teams now governs the overall testing strategy.
Human-Centric Failure Escalation
Rather than rely purely on automation, Slack designed a progressive alerting process that brings in the right humans in the right order. The system tracks test suite state as a finite state machine: two consecutive failures trigger an alert into a Slack channel dedicated to the incident.
The alert identifies the exact commit where the suite began failing, pointing to the faulty PR. From there, escalation follows a defined chain. The test triage engineer on rotation gets the alert first and investigates whether the failure stems from infrastructure or test flakiness. If not, they escalate to the suite owner, who understands the test logic. If the failure isn't from flakiness, an infrastructure change, or a service-level update, the Developer of the offending PR is brought into the conversation. If a bad commit needs removing before it reaches customers, the deploy commander is looped in to stop it.
Previously, response threads scattered across suite-owner channels, platform-owner channels, internal-tooling channels, and the deploys channel, with no one sure where the main thread lived. Now, a Slack bot reads the alerts and creates a dedicated channel for each post-merge failure. It invites the two triage engineers on schedule, exposes the Developer's name for reference, and provides an escalation button—one click adds the corresponding liaisons. From there, engineers can pull in whomever else is needed.
This human-centered process also creates better data. Consecutive-failure state tracking makes it easy to identify suites that fail often and get stakeholder attention. Recording time-to-triage and time-to-resolution for post-merge failures reveals which suites need investment. The separation of roles helps, too: product and test engineers often diagnose the same failure from different angles, and bringing a platform engineer in early can quickly distinguish a testing infrastructure incident from a code defect. During active CI incidents, Slack halts all new code deployments until infrastructure is healthy again.
Outcome: Velocity Without Added Risk
The tiered testing workflow has been in production at Slack for most of the past year. Developers now work with faster, more predictable feedback loops, and fewer end-to-end tests block merges. Fix-forward response times improved because attribution to a specific commit is immediate and escalation paths are pre-defined.
The design deliberately pushed tests into post-merge compute where infrastructure can handle them in larger, batched runs rather than demanding every suite pass synchronously for every PR. The results show that safety and velocity aren't inherently at odds—when the testing pipeline is structured so failures land in front of the correct owner by priority, code quality can be maintained with substantially less pre-merge overhead.
The change also surfaced organizational undercurrents. Slack engineers who previously operated in silos on testing concerns—product engineers, test engineers, infrastructure teams, deploy management—now share a roadmap and a common set of goals. The current trajectory for CI improvements includes better observability into bottlenecks across the pipeline and reducing reliance on end-to-end tests by building models and tooling to understand exactly which code each test platform covers.
