Why the Old EC2 Model Had to Go
Slack’s previous EC2 approach was built on continuously updating long-lived instances. Even after improving Chef rollouts with split environments, signal-based runs, and safer promotion workflows, the underlying model was still hitting hard limits. Service-level deployments remained tricky to coordinate, infrastructure drift was inevitable, and layering changes across the stack added complexity that containerized workloads didn’t have to deal with.
The answer was Shipyard, a next-generation EC2 platform that treats infrastructure as deployable artifacts. Instead of endlessly mutating instances in place, it brings service-level deployment primitives, tight integration with build and orchestration systems, and automated safety mechanisms directly to EC2.
Core Capabilities of Shipyard
Shipyard shifts responsibility away from configuration management as the center of the system. The focus moves to build pipelines, deployable artifacts, and automated safety controls. It supports both AMD64 and ARM-based Graviton instances across Ubuntu, RHEL, and Amazon Linux. This flexibility is particularly useful for workloads that cannot migrate to containers—infrastructure components, Kubernetes worker nodes, and egress network stacks.
Key features include:
- Metrics-driven deployments: Integration with Slack’s Gondola orchestration system enables progressive rollouts with metric-based safety checks. Deployments can halt automatically on service health signals or roll back to the last known good version.
- Fast provisioning: A layered image approach—a shared “golden” base image plus service-specific layers—minimizes launch-time work so instances come online quickly and predictably across regions.
- Lifecycle-based configuration: Configuration is applied during image baking and initial provisioning only, rather than continuously enforced in the background. This eliminates scheduled Chef jobs that repeatedly reapply state, reduces background load, and makes system behavior easier to reason about.
- Real-time fleet visibility: A new inventory system called Peekaboo, built on AWS EventBridge, OpenSearch, and Lambda, provides near real-time state of the entire fleet. It taps into cloud events and instance metadata, tracks even non-Shipyard instances, and offers a UI, API, and CLI.
- Automatic rotation: Instances get a limited lifespan and are rotated on a regular schedule, keeping fleets fresh and reducing the window for vulnerabilities to cause issues.
The slack-zero Foundation
At the base of Shipyard is slack-zero, a shared machine image maintained by the Compute Platform Team in collaboration with security and monitoring teams. It contains the OS baseline and hardening, networking and service discovery configuration, monitoring and security agents, and common foundational tooling. Every service inherits this standard, trusted foundation while retaining the ability to customize its own runtime on top.
The base image is immutable and ephemeral. When foundational components need updates—a security patch, monitoring change, or networking improvement—a new slack-zero image is produced, and downstream service images rebuild on top of it.
Slack builds slack-zero with AWS Image Builder rather than Packer, which provides several advantages:
- Lifecycle management: Old AMIs are cleaned up automatically with lifecycle policies, reducing storage costs.
- SSM parameter publishing: Each new image updates an SSM parameter indicating the latest available AMI, and service pipelines read this to build on the most current base.
- Event-driven automation: When baking completes successfully, EventBridge and Lambda trigger downstream pipelines in service owner accounts for dependent image rebuilds.
- Built-in testing: Before an AMI is published, Image Builder launches temporary instances and runs validation tests, verifying every image before distribution.
Service Images and the Bake/Provision Split
Service teams build their own AMIs on top of slack-zero, defining what software is installed, how the service is configured, and what happens during instance initialization. Since most configuration is baked in, instances launch consistently with minimal drift.
Shipyard deliberately separates instance preparation into two phases with different responsibilities. During the bake phase, packages are installed and configuration that is consistent across environments is included, so every instance starts from a fully prepared, known-good state. Environment-specific details—secrets, regional configuration, deployment metadata—are applied during the provisioning phase at boot. This step is intentionally lightweight: drop configuration, retrieve secrets, start services.
Moving heavy operations like package installation into the bake phase means instances can become operational in seconds rather than minutes. That speed matters for scaling events, rolling deployments, and automated instance replacement. The provisioning model balances consistency, speed, and flexibility—images deliver a stable baseline, while minimal provisioning adapts instances to their runtime environment without introducing drift over time.
Rolling Out Changes
Shipyard relies on fleets being updated through controlled replacements rather than in-place patching. Teams build a new Amazon Machine Instance (AMI) and run their deployment pipeline to roll it out, keeping the fleet consistent and predictable. For Auto Scaling Groups (ASGs), the platform uses AWS Instance Refresh; Kubernetes worker fleets use Karpenter for lifecycle-driven updates. The global deployment orchestrator, Gondola, supports these patterns and also allows services with special needs to plug in alternative rollout executors.
In an emergency, targeted configuration changes on running instances are possible through predefined AWS Systems Manager documents that run selected Chef recipes. Such changes are intended as temporary fixes only: the affected instances are marked for replacement and cycled via the regular deployment pipeline once stable.
Customer pipelines in Shipyard can have multiple stages, each representing a deployable unit—an ASG, a Kubernetes cluster, or a group of EC2 instances. Gondola updates each stage in sequence, monitors key metrics, and rolls back automatically when problems are detected, preventing issues from spreading across the fleet.
Inside a Gondola Stage
When Gondola builds an artifact, it produces a deployable package with two components:
- The AMI to roll out across the fleet
- The Chef artifact with versioned recipes tied to a Git commit
Each stage uses a service-defined executor for the rollout. For ASG-based deployments, the executor updates the launch template with the new AMI and configuration; Chef code is packaged to Amazon Simple Storage Service (S3), and new instances fetch the correct artifact and run the relevant recipes using a baked-in bootstrapper. For Kubernetes worker fleets, the executor passes the AMI and configuration metadata to Karpenter, with nodes bootstrapping the same way. Adding new executors is straightforward, so Shipyard can support different deployment models while keeping artifact handling consistent.
Who Owns What
Shipyard’s image model separates responsibilities between platform and service teams. The Compute, Security, and Monitoring teams maintain the base layer with global infrastructure components, security patches, and essential configurations. Service teams build their own AMIs on top of that base, adding service-specific software and settings.
When the Compute team ships a fix—a security patch, a monitoring agent update, or a networking change—service teams must fold the updated base into their own AMIs. Each service image therefore picks up the latest shared improvements automatically while teams focus on their respective layers.
Secrets Break the Rule
Shipyard instances are mostly immutable, with one deliberate exception: secrets. Every instance runs the Consul Template service, which can roll out updated secrets from Vault without cycling the fleet. Packages and configuration stay fixed at bake time, but credentials and certificates can be refreshed dynamically. The result is a semi-immutable design: core system and service layers remain constant, while sensitive runtime data can be updated safely as needed.
Lifecycle Enforcement with The Reaper
The Reaper evaluates two inputs to decide whether an instance should be replaced. First, it consumes signals from external systems—security tooling or AWS EC2 events—that indicate an instance has drifted from its desired state and is now “tainted.” Second, it performs periodic checks for instances that have outlived their allowed lifespan. When either condition is met, the instance is scheduled for replacement per its service policies.
This reduces configuration drift, limits exposure to vulnerabilities, and enforces redeployment over in-place modification. Even manual remote access to a production-class node generates a signal that marks the instance for eventual replacement, making the platform more auditable and predictable. The Reaper also uses Peekaboo to track instance age, so nodes past their maximum lifespan enter the same graceful replacement workflow.
Future plans include making the system more context-aware: meaningful changes such as software updates or configuration drift would trigger replacement, while read-only or low-risk actions would not cause unnecessary churn.
Controlling the Reaper
Service owners can manage how replacements occur through built-in rate limiting, scoped by service, region, or availability zone. A global pause mechanism—the “big red button”—halts all Reaper activity fleet-wide when a control object is placed in S3, providing an immediate stop during incidents. A CLI lets teams manage rate limits, inspect configuration, and activate or release the pause. Short-lived SSH certificate workflows offer a break-glass path for deeper investigation while the overall lifecycle safety posture is maintained.
Testing Before Merge
To test cookbook changes safely before merging a pull request, the Ship Quick system runs a realistic bake-and-provision test on real infrastructure. A developer runs a CLI command from the cookbook repo, where a YAML file defines the test cases. Ship Quick packages the cookbook, uploads it to S3, and queues a workflow message. A fleet of worker instances managed by a lightweight process called Longshoremen picks up the job, detaches from its Auto Scaling Group, runs the Chef workflow, streams logs back to the CLI, and terminates—unless the developer chooses to keep the instance for debugging.
Two separate worker fleets exist because of how bootstrapping works. The vanilla Ubuntu fleet bakes and tests the base slack-zero image, which cannot build on top of itself. The slack-zero fleet handles service team cookbooks that depend on the pre-baked slack-zero AMI, ensuring provisioning is validated against the same foundation used in production. Each layer is tested against its correct base. Both fleets scale automatically, and slack-zero workers are continuously updated so tests reflect the current production environment. Teams building images in their own AWS accounts can provision dedicated worker fleets and route Ship Quick jobs to them for isolation.
Extending the Platform
Shipyard has proven effective for short-lived services, and teams from the legacy EC2 platform are actively onboarding. The next challenge is long-lived instances such as data nodes, singleton services like GitHub Enterprise, or third-party business technology instances such as Atlassian JIRA. These cannot be cycled quickly, so they need safe patch and update pathways while remaining compatible with The Reaper’s lifecycle enforcement.
Work is underway with service teams to build new deploy executors in Gondola for these workloads. As Shipyard adoption grows, tooling, developer workflows, and the deployment experience will continue to evolve to serve the platform’s diverse needs.
