Why Slack moved to sandboxed, on-demand dev instances
Slack’s development environments — sandboxes where engineers test code changes before deployment, not to be confused with IDEs — live on remote EC2 instances. Each environment runs its own Slack subdomain and relies on isolated infrastructure, so experiments can't touch real users or production data. That isolation is what lets engineers iterate quickly and share work-in-progress with reviewers.
Remote over local
Slack develops remotely rather than on local machines. For a codebase as complex as Slack's, remote environments avoid the burden of local setup and reduce configuration drift: a dev environment that mirrors production is far more likely to behave like production than a personal laptop with an inconsistent toolchain. Remote instances also survive laptop crashes and make it trivial to work across multiple machines and share work with teammates. As network reliability improves, the case for remote development only strengthens.
The daily workflow
The typical flow starts by creating a feature branch and attaching it to a dev environment with slack sync-dev. That command reserves an environment and continuously syncs local edits to it via fswatch for change detection and rsync for transfer.
Frontend changes are built locally with slack run buildy:watch, which compiles assets with webpack and serves them to the dev environment over localhost. Once synced, the changes appear on the environment's subdomain for interactive testing and peer review. If the local machine goes offline, a static build can be generated on the dev environment itself so others can still view the work.
Why frontend builds moved off the instance
When Slack adopted webpack in 2017, frontend assets were built remotely on the dev environments. As the codebase grew, each build consumed an entire instance's memory. Since multiple developers shared an instance at the time, one build would interrupt everyone else on it. Moving webpack to local machines solved the problem. Today, with one dev environment per instance and more powerful hardware, remote builds would be feasible again and would smooth the developer experience. But the current approach remains fast and scalable, so Slack hasn't changed it.
Tooling grows from necessity
Early on, copying changes to a dev environment was done manually — slow and error-prone. That pain drove the creation of slack sync-dev and, since then, more than 60 internal command line tools. Examples include slack bot-me for creating a bot user on the current environment and slack tail-dev for tailing remote logs.
Scaling from one shared sandbox to on-demand instances
In 2014, Slack had a single shared dev environment. One person's mistake could block everyone else. By the end of 2019, that had grown to 550 environments, enough for every engineer to claim a separate one.
But the number of environments per EC2 instance fell over the same period. Sharing an instance meant one developer's memory-heavy process could degrade performance for everyone else, so Slack isolated environments from one another.
That isolation came at a cost: more instances and significant engineering time spent manually provisioning and deprovisioning statically managed hardware. Long-lived instances also degraded reliability over time. Slack responded by building a demand-based provisioning system that spins up instances when needed and automatically tears them down when developers finish. That shift reversed the growth trend in the first chart — instead of keeping hundreds of instances running concurrently, Slack now uses capacity more efficiently. The engineering team plans to cover those scaling changes in more detail in a future post.
