When the Backup Is Older Than the Outage
On January 29, 2024, an internal Kibana cluster at Slack began failing because its nodes ran out of disk space. The root cause traced back to an earlier architectural decision: the Elasticsearch instance backing Kibana was colocated on the same hosts as the Kibana application itself, coupling storage and application lifecycle on shared nodes. As those nodes degraded, engineers lost access to the custom dashboards and visualizations they relied on to monitor application performance.
The cluster deteriorated past the point of recovery, forcing a full rebuild. The team planned to restore Kibana objects from backup after cycling in fresh hosts, but the most recent viable backup was nearly two years old. The backup and restore process had been configured once and never revisited—no alerts were in place to signal when it stopped working. The runbook was outdated, and the old backup failed during the restore attempt. Internal links and visualizations were lost, and indexes and index patterns had to be recreated by hand.
The lesson was uncomfortable but clear: processes, procedures, and runbooks that aren't actively exercised will silently rot and fail exactly when they're needed most. Incident response restores service quickly; what happens afterward determines whether the incident becomes a liability or an opportunity.
Planned Destruction as a Testing Strategy
Post-incident work focused on preventing a repeat: scheduled backup scripts for Elasticsearch clusters in every environment, runbooks corrected based on what actually happened, and verification that Amazon S3 retention policies were correct. But the team wanted to prove the fixes worked under realistic conditions, not just on paper. The idea that emerged was unconventional: deliberately break a development Kibana cluster and run the new backup and restore process against it end to end.
The development cluster mirrored production configuration closely enough to provide a meaningful test. The plan specified which cluster would be broken, how it would be broken, and how restoration would proceed.
Executing the Exercise
The event was scheduled for a quiet Thursday morning, and the whole team was invited. The team filled the disk on the Kibana nodes, watched the cluster fail in real time, and confirmed alerts triggered as designed. Working through the updated runbook, they cycled the cluster through a complete rebuild. The system recovered successfully from the staged incident.
Recovery worked, but not within the team's target of one hour. Several commands in the runbook were poorly understood and difficult to parse in a stressful situation. Copy-pasting was itself a challenge due to formatting problems. The backups did fully restore the cluster's state, and the exercise surfaced firewall rules that needed to be added to the infrastructure as code—an unexpected discovery that saved future headaches.
A second test followed: migrating the general development Kibana instance and its Elasticsearch cluster onto Kubernetes. This gave the team a chance to run the improved backup script against a high-use cluster. With better-runbook comprehension and updated provisioning scripts, the migration completed with roughly 30 minutes of downtime.
Both exercises exposed minor gaps in the runbooks and restoration process, which the team corrected as they went. The experience also motivated a larger improvement: the scheduled backup script was expanded into a full-featured CLI backup and restore program, capable of restoring a Kibana backup from cloud storage with a single command.
Finding the Failure Mode Before It Finds You
Every production system is broken in some way that hasn't surfaced yet. Deliberate, scheduled chaos is a practical way to find those hidden faults before a real incident does. Load testing before customers hit a bottleneck, turning services off to simulate unexpected outages, and upgrading dependencies on a regular cadence are all forms of the same idea. Routine maintenance often feels dry until an incident makes it urgent.
Planned opportunities to break things make that maintenance tangible and memorable. The exercise put the team in realistic situations they wouldn't have encountered through traditional testing, and each iteration of breaking and restoring sharpened both the tooling and the team's understanding of it. The investment paid off during the Kubernetes migration, where improved processes turned what could have been a prolonged outage into a half-hour blip.
The takeaway: break your own systems on purpose. Restore them, then do it again. The process and tooling will be better for the next time you need them under pressure.
