Deploy Safety: Reducing customer impact from change

Slack’s Deploy Safety Program: Cutting Customer Impact by 90%

By early 2025, Slack had reduced customer impact hours from change-triggered incidents by 90% compared to the peak seen in 2024. That improvement is the result of the Deploy Safety Program, an initiative that began in mid-2023 to overhaul how the company approaches deployments across its many internal services.

The problem was clear from internal data: Slack had become increasingly mission-critical for customers, and a growing majority—73%—of customer-facing incidents were triggered by Slack-induced change, most notably code deploys. These incidents spanned a wide variety of systems and deployment processes, making them hard to address with point fixes to individual services.

Past reliability efforts often zeroed in on a single deploy system or service. That approach led to manual change processes that slowed innovation, dampened engineering morale, and made it harder to meet business goals. The company needed a strategy that worked across all deployment methods for its highest-importance services.

North Star Goals and the Deploy Safety Manifesto

The initial goals focused on three areas for high-importance services:

• Reducing impact time from deployments: automated detection and remediation within 10 minutes; manual detection and remediation within 20 minutes.
• Reducing severity of impact: detect problematic deployments before they reach 10% of the fleet.
• Maintaining development velocity: safety improvements should not bog down the pace of change.

These goals evolved into a broader Deploy Safety Manifesto applicable to all of Slack's deployment systems and processes. The manifesto pushes for automated system improvements, safety guardrails, and cultural changes.

Choosing a Metric That Matters

Programs need measurement, but choosing the right metric was a constant source of discussion. The team needed something that served as a reasonable analog for customer sentiment, not just an engineering output.

The chosen Deploy Safety metric is:

Hours of customer impact from high severity and selected medium severity change-triggered incidents.

The "selected" qualifier exists because Slack's severity levels reflect current or impending impact, not the final impact that requires post-hoc analysis. Medium severity incidents were filtered based on a relevant level of actual customer impact.

Key criteria guided metric design:

• Measure results, not just activity.
• Understand what is measured—the real thing versus the analog.
• Maintain consistency, especially for subjective parts of the measurement.
• Continually validate that the metric tracks customer sentiment, using direct feedback from leaders who talk to customers.

Investment Strategy: Bias for Action

At the start, there were two large unknowns: which projects would have the highest likelihood of success across multiple incident sources, and when each project would deliver measurable impact. Incident data is a trailing dataset—there is a time delay before results appear—while customers were feeling pain immediately.

The investment strategy reflected those realities:

• Invest widely at first and bias for action.
• Focus on areas of known pain first.
• Double down on projects or patterns based on results.
• Curtail investment in the least impactful areas.
• Keep a flexible, shorter-term roadmap that can change based on findings.

Projects were chosen to influence one or more of: earlier detection during deployment, faster automatic remediation, faster manual remediation, or reduced severity through blast-radius control.

The Webapp backend was identified as the largest source of change-triggered incidents. The investment flow there illustrates the pattern: a quarter of work to engineer automatic metric monitoring, another quarter to confirm customer-impact alignment, then investment in automatic deployments and rollback. After proving success with automatic rollbacks keeping customer impact under 10 minutes, the team invested further in additional metric monitoring and manual rollback optimizations, plus a manual Frontend rollback capability.

That work fed into Slack's centralised deployment orchestration system, inspired by ReleaseBot and AWS Pipelines, to unify metrics-based deployments with automatic remediation beyond Slack Bedrock and Kubernetes into many other deployment systems. Webapp backend, frontend, and a portion of infrastructure deployments are now significantly safer, with quarterly improvement.

Not every project delivered the desired impact. Some were less successful than others, but the team notes these are not failures—they are critical inputs that guide investment and reveal which areas provide greater value. That variability is by design.

Results and the Lag Problem

The chart tells a story of non-linear progress. After the first quarter of work, improvement appeared even before most changes were deployed—simply communicating the program's existence helped. The peak quarter of impact came between February and April 2024, after projects started delivering changes. The team had confidence the work would pay off based on early results from metrics-based deploy alerts with manual remediation at the Webapp backend; the peak would have been higher without that work. What was needed was automatic, not manual, remediation.

Once automatic rollbacks were introduced, results improved dramatically. There is a consistent 3-6 month lag between delivering work and observing its full impact. Because the metric relies on trailing incident data, targets were set based on expected results from work delivered in the previous quarter, not on immediate measurement of newly shipped work. The target for 2025 was reduced again, focusing on mitigating infrequent spikes through deployment system consistency.

Lessons Learned

The program required continual learning. Several lessons proved critical:

• Prioritisation and alignment matter. Executive reviews every 4-6 weeks kept the program aligned and supported. Deploy Safety was embedded in company and engineering goals, with active sponsorship from senior leadership.
• Patience with trailing metrics is essential. When measurement lags months behind delivery, you need faith in the process and the agility to change course once results confirm or contradict your assumptions. Meanwhile, track intermediate signals like issue detection to know the improvement is at least functioning.
• Adoption lags familiarity. Engineers delayed using new processes until comfortable, worried they would make things worse. Direct, repeated training—"Just roll back!"—and frequent use of rollback tooling during incidents built the fluency required to make the tools routine. Infrequent use is as good as not building the capability.
• Direct outreach to engineering teams is critical. The program team engaged directly with individual teams to understand their systems, provide guidance, and encourage innovation. Teams differ: some know their pain areas and have ideas; others want to improve but need additional resources.
• Keep the top-line metric consistent. Pick a metric, stay consistent, and refine based on validation. There is no perfect metric, and time spent deliberating is time not spent improving.
• Communicate consistently with all engineering staff. Management was well aligned, but the rationale and results were not always clear to general engineers—an opportunity for better alignment remains.

Future Directions

Reliability is a cornerstone of trust, and Slack intends ongoing investment in Deploy Safety. Planned work includes:

• Further improvements in automatic metrics-based deployments and remediation.
• Greater consistency in Deploy Safe processes across all deployments to mitigate unexpected spikes in customer impact.
• Expanding the program's scope to migrate remaining manual deploy processes to code-based deploys using Deploy Safe practices.

Ongoing projects in this space include expanding centralised deployment orchestration to other infrastructure patterns such as EC2 and Terraform, automatic rollbacks for Frontend, metric quality improvements, AI-based anomaly detection, and broader rollout of AI-generated pre-production tests.