Making Chef Deploys Safer Without Rebuilding Everything

Slack's fleet runs on Chef, and as the infrastructure has grown, the team has hit a familiar problem: how to add safety rails to a configuration management system that was never built for rapid, staged rollout. The brute-force answer would have been to migrate to Chef Policyfiles, but that would mean replacing roles and environments across dozens of teams—a significant effort with near-term risk that outweighs the long-term benefit. Instead, the team improved the existing EC2 framework in a way that maintains compatibility with existing cookbooks and roles.

Bucketizing the Production Environment

Chef runs on Slack instances used to happen via a cron job every few hours, primarily for compliance. The timings were staggered across availability zones to avoid hitting the entire fleet at once, but there was a gap: newly provisioned nodes pulled the latest configuration from a single shared production environment immediately upon boot. During large scale-out events, dozens or hundreds of nodes could start up with a broken configuration before anyone noticed.

The fix was to split the single production Chef environment into six buckets: prod-1 through prod-6. Service teams still launch instances as "prod," but a behind-the-scenes mapping assigns each instance to a numbered environment based on its Availability Zone. New nodes are now distributed across isolated environments instead of pulling from one global source of truth.

At boot time, the Poptart Bootstrap tool runs via cloud-init—creating the Chef object, setting up DNS entries, and posting status messages to Slack. This tool was extended with logic that inspects a node's AZ ID and assigns it to the corresponding numbered production environment. Cookbook promotions can now be made per environment, so a bad change only affects nodes in specific AZs rather than the entire fleet. When no updates are in progress, rollouts can complete more quickly.

Fresh changes flow into sandbox environments at the top of the hour via a Kubernetes cron job, then to dev environments, and finally into production starting at 30 minutes past the hour. prod-1 acts as a canary that gets updated every hour to catch production-impacting issues early. prod-2 through prod-6 run on a release train: before a new version reaches prod-2, the previous version must have progressed through prod-6. This incremental rollout gives the team a chance to catch regressions before they spread.

The tradeoff—longer rollout times across the fleet—buys a substantial safety margin. Staggered updates between AZs create time windows to catch problems, and the team can keep scaling safely in unaffected AZs while a fix is developed.

Replacing Cron with Event-Driven Triggers

Fixed cron schedules break down when multiple environments receive updates at different times, as the next version must wait for the current rollout to finish across all environments. Slack replaced scheduled runs with Chef Summoner, a service that triggers Chef runs based on signals, running only when updates are actually available.

The rollout pipeline runs through Chef Librarian, which watches for new cookbook artifacts, uploads them to all Chef stacks, and promotes specific versions to environments via an API endpoint. Librarian now also sends a message to an S3 bucket each time it promotes a version. The S3 structure groups by Chef stack, then by environment within each stack, with a JSON object under each environment key:

{
  "Splay": 15,
  "Timestamp": "2025-07-28T02:02:31.054989714Z",
  "ManifestRecord": {
    "version": "20250728.1753666491.0",
    "chef_shard": "basalt",
    "datetime": 1753666611,
    "latest_commit_hash": "XXXXXXXXXXXXXX",
    "manifest_content": {
      "base_version": "20250728.1753666491.0",
      "latest_commit_hash": "XXXXXXXXXXXXXX",
      "author": "Archie Gunasekara <[email protected]>",
      "cookbook_versions": {
        "apt": "7.5.23",
        ...
        "aws": "9.2.1"
      },
      "site_cookbook_versions": {
        "apache2": "20250728.1753666491.0",
        ...
        "squid": "20250728.1753666491.0"
      }
    },
    "s3_bucket": "BUCKET_NAME",
    "s3_key": "20250728.1753666491.0.tar.gz",
    "ttl": 1756085811,
    "upload_complete": true
  }
}

Chef Summoner runs on every node, reads the S3 key for its stack and environment, and schedules a Chef run when a new version shows up. A splay value staggers runs so nodes in the same environment don't all execute at once, and the value can be adjusted for different situations, like spreading runs more intentionally after a custom signal.

Even with no new artifacts to promote, Summoner still triggers Chef at least once every 12 hours for compliance and to keep nodes in their expected state. Summoner stores its own state locally—last run time and artifact version—to compare against new signals and decide if a Chef run is needed.

A Safety Net for the Safety Net

Since Summoner now handles all routine Chef runs, a broken Summoner version could stop Chef entirely, making it impossible to push out a fix through the normal path. Each node carries a fallback cron job that checks Summoner's local state; if Chef hasn't run in 12 hours, it triggers Chef directly. This creates a recovery path for rolling out a working Summoner. The team also has tooling to trigger ad hoc Chef runs across the fleet when needed.

Maintenance Mode for a Legacy Platform

With these changes, EC2 infrastructure rollouts are notably safer. But one gap remains: service-level deployments still aren't practical. Creating dedicated Chef environments for each service would be unmanageable with hundreds of services at Slack, even though artifact promotion per service is theoretically possible.

The EC2 platform is now marked as feature complete and in maintenance mode. In its place, Slack is building Shipyard—a new EC2 ecosystem designed for teams that can't move to the container-based Bedrock platform. Shipyard is intended as a ground-up redesign with service-level deployments, metric-driven rollouts, and automated rollbacks. A soft launch is planned this quarter with two initial teams.