How Slack ships code to production
Slack’s deployment pipeline is built around a staged rollout that balances developer velocity against the risk of shipping broken code to customers. Code merged to master isn’t deployed immediately. Instead, releases go out during North America business hours, roughly 12 times per day, when engineering staff are available to respond to issues. A designated deploy commander oversees each release, watching metrics and coordinating with engineers if something goes wrong.
Release branch creation
Every deploy starts with a new release branch in Git. This branch provides a point for tagging builds, and it becomes the place where hotfixes are cherry-picked in when problems are discovered during rollout.
Staging, dogfood, and canary
The new build first goes to staging servers, which run a production configuration but don’t accept public traffic. An automated smoke test runs there, followed by manual testing. From staging, the release moves to the dogfood tier, a set of hosts serving Slack’s own internal workspaces. Because Slack employees are heavy users of the product, dogfooding surfaces many issues before external customers see them.
Once dogfood looks clean, the build is deployed to canary, where approximately 2% of production traffic is routed. If error rates remain stable and no alerts fire, rollout proceeds in percentage increments of 10, 25, 50, 75, and 100% of production traffic.
Handling failures
If a release causes problems, the deploy commander works with the responsible engineers to identify the offending pull request and revert it. The revert is cherry-picked into the release branch, and a new build is made. When the issue reaches production users before it’s caught, the immediate step is to roll back to the last known-good build to restore service. Investigation follows once service is restored.
Infrastructure shifts that enabled the rollout model
Moving from push to pull
Slack’s early infrastructure was small enough that a deploy was a simple rsync to all servers, with no intermediate production tiers between staging and full rollout. In that era, any engineer could push code on their own schedule. That stopped working as the number of servers grew; each new host made push-based deploys slower, and even parallel rsync strategies hit their ceiling.
Slack replaced pushes with a parallel pull-based system. Each server is signaled through a Consul key change and pulls the new build concurrently. This keeps deploy times stable as the fleet grows.
Making deploys atomic
Another problem surfaced as files were copied onto production servers: the process wasn’t atomic. Call sites for new functions could land before the functions themselves existed, causing a burst of internal errors in API requests and broken pages during every deploy.
Slack fixed this by keeping hot and cold directories. Production traffic is served from the hot directory while the cold directory is prepared with the new release. The server is drained of active processes, and then the switch between directories happens instantaneously.
When speed started hurting stability
By 2018, the emphasis on deploying as quickly as possible was degrading product stability. Slack had the infrastructure systems in place to move fast, but the rollout process needed to change as the company grew and the product became more mission-critical. Production rollout is now deliberately gated through the tiers described above, supported by better monitoring and tooling that can catch bugs with minimal user impact.
The deploy infrastructure itself remains built on fast, atomic deploys. Both systems continue to evolve as Slack works on further tooling and automation around the release process.
