From One Stack to Many: Rethinking Chef at Scale
Slack runs tens of thousands of EC2 instances across a fleet that includes Vitess databases, Kubernetes workers, and core application components. Most instances run Ubuntu; a portion run Amazon Linux. Provisioning and change deployment rely on internally-built services with Chef at the center. As that fleet grew, the original single-stack Chef design became a liability, and replacing it with a sharded architecture surfaced several engineering challenges worth examining.
The Original Setup and Its Bottlenecks
Early on, Slack operated one Chef stack backed by EC2 instances, an AWS application load balancer, an RDS cluster, and an AWS OpenSearch cluster. The stack scaled vertically and horizontally as demand increased. Three environments — Sandbox, Dev, and Prod — mapped directly to fleet nodes.
Cookbooks fell into two categories: those pulled from the Chef Supermarket and those authored internally. A process called DishPig handled uploads. After a code merge, a CI job detected changes to cookbooks or roles, built an artifact, pushed it to an S3 bucket, and notified an SQS queue. DishPig ran hourly, pulling the artifact and uploading changed cookbooks to the Chef server. Supermarket cookbooks kept their own versions, but internally created cookbooks were always uploaded with a fixed version number, meaning only one version ever existed on the server.
All three environments received updated Supermarket cookbooks immediately after upload. Since internal cookbooks kept identical version references, their environment entries needed no changes. The result: every environment picked up changes at the top of the hour.
Developers had tooling to spin up temporary Chef environments with specific cookbook versions, allowing testing against a small subset of nodes. Once testing finished, the tooling cleaned up both the environment and the custom cookbook versions.
That approach carried two serious drawbacks. First, every change landed in all environments at once, so a faulty cookbook could disrupt Chef runs and new server provisions across the entire fleet. Second, a single Chef stack was a critical single point of failure; any problem with that stack rippled outward.
Sharding the Chef Infrastructure
Eliminating the single point of failure meant running multiple Chef stacks. Spreading load across shards also made it possible to direct new provisions to healthy stacks if one failed. But sharding introduces its own problems.
Assigning Nodes to Shards
An AWS Route53 Weighted CNAME record became the assignment mechanism. On boot, an instance queries the record and, based on the assigned weight, receives a record pointing to a specific Chef stack. The architecture also separates development and production Chef infrastructure into distinct stacks, strengthening the boundary between environments.
Node Discovery Without Chef Search
Slack historically lacked a dedicated EC2 inventory system, so Chef served that role. Teams queried the Chef server directly to find nodes by role, region, or other criteria. With a single stack, Chef search returned a complete picture. Sharding broke that: a query against one stack only returned nodes assigned to that stack.
The problem runs deeper because of how a Chef run executes. A Chef run happens in two phases: compile and converge. During compile, Chef evaluates run lists, reads cookbooks, sets attributes and variables, and maps out resources. During converge, Chef actually creates those resources. Some Slack cookbooks built attributes and variables based on Chef search results during compile.
To replace node discovery, Slack turned to Consul, which already existed internally. Services started registering with tags for service discovery. Queries against Consul replaced Chef searches for node information.
But that introduced a circular dependency: Slack uses the Nebula overlay network, which Chef configures. Querying Consul requires Nebula connectivity, yet Nebula setup depends on Chef. Using Consul values during the Chef compile phase was therefore impossible — resolving the dependency required values that weren't available yet.
Slack's answer was Chef's ruby_block resource. The resource executes during the converge phase, not compile. By placing node-lookup logic inside ruby_block resources and controlling execution order, Nebula gets installed and configured before those blocks run. For values computed inside Ruby blocks that other parts of the Chef run need, the lazy keyword forces evaluation during converge instead of compile.
Beyond that groundwork, Slack built a set of Chef library functions that perform node lookups by tags, service names, and other criteria — replicating the previous Chef search functionality across Consul. Example:
node.override['some_attribute'] = []
# In Chef, when a resource is defined all its variables are evaluated during
# compile time and the execution of the resource takes place in converge phase.
# So if the value of a particular attribute is changed in converge
# (and not in compile) the resource will be executed with the old value.
# Therefore we need to put the following call inside the ruby block because,
# We need to make sure Nebula is up so we can connect to Consul (Nebula gets stup during converge time)
ruby_block 'lets_set_some_attribute' do
block do
extend SomeHelper
node.override['some_attribute'] = consul_service_lookup_helper_func()
end
end
# Please note that we are not using `lazy` with the `only_if` below for the value that is calculated in the ruby block above
# It's because `only_if` is already lazy
# https://github.com/chef/chef/issues/10243#issuecomment-668187830
systemd_unit 'some_service' do
action [:enable, :start]
only_if { node.override['some_attribute'].empty?
endThe tradeoff is heavier, more complex Ruby logic embedded in some cookbooks. But it allowed the sharded rollout to proceed — and it sidesteps the compile-time discovery problem entirely.
Searching a Sharded Fleet: Gaz and Shearch
Because Chef had been the de facto inventory system, the original single-stack design let developers and tooling track node attributes and feature rollout progress through Chef searches. The internal interface Gaz provided Chef search and reporting to developers.
Sharding meant Gaz could no longer ask one Chef stack for a fleet-wide answer. Slack built Shearch (Sharded Chef Search), an API that accepts Chef queries, runs them against every shard, and merges the results. Gaz sits behind Shearch instead of reaching into a Chef stack directly.
Developers also replaced Chef Knife on their machines with Gnife, a Go-based tool that exposes the same functionality but queries all shards through Shearch. Numerous internally-developed tools and libraries required updates to call Shearch instead of a Chef endpoint — a significant undertaking.
Sharding improved resilience considerably, but cookbook deployment was still synchronized: all environments updated at once with identical cookbook versions.
Cookbook Upload Consistency Across Shards
To move quickly, Slack kept DishPig but deployed multiple instances — one per Chef stack. The upload pipeline changed as well. The S3 bucket now sends notifications to an SNS topic, which fans out to multiple SQS queues, each tied to a different DishPig deployment. That lets each Chef stack receive cookbook changes independently, but without any consistency checks between stacks.
That independence caused drift. Artifacts built just before the hourly DishPig trigger generated SQS messages that some queues received in time while others missed. The result: Chef stacks ended up running different cookbook versions. A small monitoring system was built to detect and alert on consistency errors, but it was far from a robust solution.
Versioned cookbooks with Chef Librarian
Cookbook updates previously meant environments across all Chef stacks changed at once, with no way to track which versions of the code were running where. To close that gap, we replaced our delivery service DishPig with a new service called Chef Librarian.
All cookbooks live in one Git repository. Internally developed cookbooks are under site-cookbooks, and Supermarket cookbooks live under cookbooks.
. ├── cookbooks │ ├── supermarket-cookbook-1 │ │ ├── files │ │ │ └── default │ │ │ └── file1.txt │ │ └── recipes │ │ └── default.rb │ └── supermarket-cookbook-2 │ ├── files │ │ └── default │ │ └── file1.txt │ └── recipes │ └── default.rb ├── site-cookbooks │ ├── our-cookbook-1 │ │ ├── files │ │ │ └── default │ │ │ └── file1.txt │ │ └── recipes │ │ └── default.rb │ └── our-cookbook-2 │ ├── files │ │ └── default │ │ └── file1.txt │ └── recipes │ └── default.rb └── roles ├── sample-role-1.json └── sample-role-2.json
When a change is merged, a GitHub Actions build job creates a tarball containing the complete repository. For internal cookbooks, the job bumps the version number to the standard Chef format YYYYMMDD.TIMESTAMP.0 (for example, 20240809.1723164435.0); Supermarket cookbook versions stay fixed unless we pull down a new upstream release.
The built artifact is then uploaded to S3, which emits a message into an SQS queue. Chef Librarian listens on that queue, and once a new artifact arrives it uploads the new cookbook versions to all Chef stacks. The versions are available but not active until an environment explicitly points at them. Even if a cookbook didn't change, it gets re-uploaded under the new version.
Chef Librarian exposes two REST endpoints to control rollouts:
/update_environment_to_version– Points a target environment at a specific artifact version./update_environment_from_environment– Sets an environment's versions to match those of another environment.
These endpoints allow us to promote an artifact to sandbox and development first, watch metrics for regressions, and only then move it to production. A bad change no longer fans out to every environment at the same time, and the blast radius of a rollout problem is much smaller.
Artifact versions, their deployment targets, and environment state are recorded in DynamoDB, giving us a clear view of what is running where.
Chef roles are not versioned, which makes them riskier: a role upload propagates everywhere at once. To make role changes safer, we've stripped most logic out of our roles and left them with just core attributes and a run list. We've also split Chef into dev and prod stacks and only push roles to the relevant shards when the matching environment is updated (for example, role updates go to prod shards only during prod environment promotion).
Environment promotion is currently driven by a Kubernetes CronJob that calls the Chef Librarian API on a schedule. Before promoting, the job checks a designated S3 bucket for a block key; if failures are detected in sandbox or dev environments, an operator places that key in the bucket to halt the rollout. This is still a manual safety interlock, and we're evaluating replacing the CronJob with a system that watches metrics and triggers the block automatically without human action.
The Chef Librarian interface exposes audit history: when an artifact version was uploaded, which commits are in it, and when an environment was moved to a given version on a given Chef shard.
A Slack app built on the same service notifies the engineer responsible for a change when it is promoted. The app maps the Git commit author to a Slack handle via the Slack API and mentions them directly in the promotion message.
Next steps
Deployment safety improved, but we are not done. One option under evaluation is splitting production Chef environments along AWS availability zone boundaries so that each zone can be promoted independently, further containing any bad rollout.
We're also investigating a longer-term migration to Chef PolicyFiles and PolicyGroups. That's a substantial architectural departure from our current setup and would give us more granular control over which nodes receive which changes. Given our fleet size, the migration is non-trivial and still in the research phase.
Look for more details from our team as these projects move forward.
