Availability Incident on October 9
GitHub recorded a single availability incident in October 2020. For two hours and 32 minutes starting at 21:30 UTC on October 9, users experienced degraded state and significant impact on issues, pull requests, webhooks, GitHub Actions, and GitHub Pages.
The root cause traced back to routine ZooKeeper upgrades. While reprovisioning nodes, new hosts were added faster than intended, which led to a second leader being elected. The result was a logically separate second ZooKeeper cluster alongside the original one.
In that state, a single Kafka broker in the cluster backing GitHub’s internal background job system connected to the newly formed second ZooKeeper cluster and elected itself as the Kafka controller. With two distinct Kafka clusters now serving conflicting cluster state to clients, roughly 10% of write requests to the background job service failed. That triggered a cascade of queued jobs as traffic and worker capacity shifted to the secondary job processing system.
No jobs were permanently lost. Queue backups were significant for some systems, but client-side retry behavior and redundant queueing systems kept the impact contained.
Preventive Measures
To avoid a recurrence, GitHub has updated its ZooKeeper provisioning checklist and is planning automation to handle ZooKeeper and Kafka cluster maintenance going forward. Further engineering context is available in the new Building GitHub blog series, which covers how teams across the organization are improving internal tooling and infrastructure.
