A build platform that can keep up
Slack’s build platform has grown steadily since the early days when Jenkins was introduced for continuous integration and delivery. As Slack scaled and more teams relied on Jenkins for their builds, each team began running its own instance with unique plugins, credentials, security practices, and management processes. That independence let teams move quickly in the beginning, but it eventually produced a sprawl of hard-to-maintain “snowflake” clusters, each with its own infrastructure quirks, plugin versions, and vulnerabilities to deal with.
The cost of that fragmentation wasn’t just an occasional disruption. In practice, a poorly designed build system causes chronic day-to-day friction. Engineers at Slack ran into missing immutable infrastructure that hurt consistency and debuggability, manually added credentials that couldn’t be reproduced later, poor resource utilization from static EC2 Jenkins agents, accumulating technical debt, and business and deployment logic jammed into a single place. Standard strategies for backup, disaster recovery, observability, logging, and tracing were missing across the board. Upgrading or redeploying Jenkins clusters was risky because the clusters were not stateless, so a mundane update could easily produce high-severity warnings or an incident.
From a business perspective, these engineering problems surfaced as lost developer productivity, incidents triggered by configuration changes like ssh-key rotations or software upgrades, reduced person-cycles for operations work, and inefficient use of CPU and memory on underutilized Jenkins servers. Build history was lost during downtime, SLA/SLOs were hard to define with so little control over the service, and running Jenkins around the clock—even during maintenance windows—wasn’t feasible.
Making modernization incremental
The Build team’s task was to fix the platform without breaking what already worked. An outright replacement was off the table for several practical reasons: the existing system was functional, some build scripts sat in the critical path of deployment, the infrastructure was tightly coupled to the Jenkins ecosystem, and a full migration would have consumed more resources than modernizing what was already in place.
So the team designed a prototype around a “build as a service” platform—a shared foundation that teams could tune for their needs, built on the learnings from research into how other large-scale companies run their build systems.
Key design choices in the prototype
Stateless, immutable CI service
The most fundamental change was separating business logic from underlying build infrastructure. Build-related scripts were moved into a repository independent from the business logic, and Kubernetes was introduced to manage the Jenkins services. That shift solved problems around immutable infrastructure, resource efficiency, and high availability while eliminating residual state—every service build started from scratch, which made deployment safer and made it easier to shift quality checks earlier in the process. The maintainability win was significant because the CI service no longer carried state between runs.
Agent options: ephemeral and static
Users could pick from two types of build agents. Ephemeral agents ran on Kubernetes workers: they executed the build job, then terminated when the job finished. Static agents ran on AWS EC2 machines, remained available after job completion, and served as an incremental step forward while more teams transitioned to the ephemeral model, which demands more testing and effort.
Security baked into the pipeline
Security wasn't a review after the fact. Each time the Jenkins service was built, vulnerability scanning was part of the pipeline, and IAM and RBAC policies were configured per cluster from the start so that access control and secrets were consistent and auditable.
Shift-left testing to catch issues early
A blanket test cluster and a pre-staging area were introduced for validating small- or large-impact changes before they reached broader staging environments. High-risk changes could remain “baking” for extended periods before pushing to production, and teams could add extra stages as needed. Tooling also supported local debugging, so many issues could be caught before infrastructure code was deployed.
Standardization and GitOps
Central to the design was standardizing configuration so that a single fix could be applied uniformly to every Jenkins instance. The configuration-as-code plugin (casc) was used to manage the entire Jenkins controller with a single YAML file—covering credentials, security matrices, and other settings. The team also coordinated closely with the casc plugin’s open source project. Central storage ensured every Jenkins instance used shared plugins, allowing automatic upgrades without manual intervention or incompatible version hand-wringing.
State management was handled through AWS EFS, needed for things like build history and configuration changes. Backups ran automatically at regular intervals with rollback functionality built in for disaster recovery.
Nothing ran “live” on the controllers—the platform enforced GitOps-style state management. Manual changes weren’t possible; all changes were pulled from Git, making it the single source of truth and enabling safe recreation of the entire infrastructure from scratch. Jinja2 templates made it easy for users to build new clusters by reusing existing configurations and sub-configurations.
Each cluster was outfitted with tools for metrics, logging, and tracing from the get-go: Prometheus handled metrics, the ELK stack was used for log tracking, and Honeycomb was used for tracing. Centralized credential management facilitated credential reuse across clusters. Everything—Jenkins, OS, packages, plugins—was packaged into a container Dockerfile, so upgrading was quick and consistent.
Ownership and collaboration
Service owners were given complete control over when to build and deploy their service, and that workflow could be configured to trigger automatically on pushed commits if they wanted. For cases where a move to Kubernetes wasn’t immediately possible, the prototype supported “containers in place” as a halfway step.
Before committing to the design, the Build team shared the plan in Slack’s company-wide design review, opening it to feedback from developers and experts across departments. That exposed a wider range of use cases, surfaced real consequences for service teams, and sharpened the team’s approach to scaling the build platform in a way that would support Slack’s growth for years to come.
Ownership Without a Bottleneck
Rather than concentrating every system under the Build team, Slack adopted a distributed ownership model. The Build team manages the core build platform infrastructure, while service-owner teams manage the systems that run on top of it. The diagram below illustrates how responsibilities are split.
What the Business Got Out of It
The shift produced several concrete benefits. Most notably, time to market shrank: individual services could be built and deployed both quickly and securely. The time needed to address security vulnerabilities dropped significantly, and standardizing the Jenkins inventory eliminated the many divergent code paths previously required to maintain the fleet. The metrics below summarize the results.
Infrastructure changes also became faster to roll out — and just as fast to roll back when something went wrong.
Hurdles and Lessons Learned
Moving to new technology across existing infrastructure was not without friction. The team encountered a range of challenges and took away several key lessons:
- Familiarity with Kubernetes was a prerequisite, and the Build team had to educate other teams as needs arose.
- Documentation quality had to be excellent for other teams to take ownership of their infrastructure.
- Ephemeral Jenkins agents were particularly difficult to introduce. They required reverse engineering the existing EC2 agents and reimplementing them, which was time-consuming. The solution was incremental: first move Jenkins controllers to Kubernetes, then move the agents.
- A rock-solid debugging guide was essential, since debugging in Kubernetes differs greatly from working with EC2 instances.
- Engaging with Jenkins' open source community proved valuable. Live chats, such as this one, provided quick answers on how others solved similar problems.
- Migrating production services required extreme care, especially for those critical to Slack's uptime. The team stood up the new infrastructure with harmonized configurations so teams could test confidently, repointed endpoints only after stakeholders validated their workflows, and kept the old infrastructure on standby behind non-traffic-serving endpoints for fast rollback.
- Regular training sessions shared learnings across the organization.
- Existing build scripts could be reused in the new environment, so users weren't forced to learn something new without a real need.
- Working closely with user requests — triaging issues and processing migrations — built trust with the community, who in turn contributed features they found impactful.
- Adopting a GitOps mindset was initially difficult due to older habits.
- Metrics, logging, and alerting were critical for managing clusters at scale.
- Automated tests ensured correct processes were followed as the number of users grew.
The rollout began by migrating a few existing production build clusters to the new method, which generated valuable feedback for the team. All new clusters were built on the proposed system from the outset, accelerating delivery of important features. Migration of all services is still in progress, alongside work on features that will reduce manual maintenance and automation tasks.
Looking ahead, Slack intends to offer build-as-a-service to MLOps, SecOps, and other operations teams. That would let those groups focus on business logic rather than infrastructure, further improving time to market.
