DNS failure disrupts GitHub services for nearly 20 hours
GitHub recorded a single availability incident in October 2024, causing broad degradation across multiple products over a roughly 19-hour stretch.
What happened
The trouble began on October 11 at 05:59 UTC, when the DNS infrastructure in one of GitHub's sites stopped resolving lookups after a database migration. Recovery attempts triggered cascading failures that further destabilized the site's DNS systems. Customers first noticed problems around 17:31 UTC.
The blast radius was wide. Four percent of Copilot users saw degraded IDE code completions, and 25% of Actions workflow users dealt with delays exceeding five minutes. For a roughly four-hour window, all code search requests failed.
Response and recovery
At 18:05 UTC, engineers tried repointing the degraded DNS site to another location. That move restored connectivity within the affected site but broke traffic from healthy sites back to it, forcing a change of plans.
By 20:52 UTC, the team had settled on a new remediation approach and began deploying temporary DNS resolution capabilities to the degraded site. DNS resolution started recovering at 21:46 UTC and reached full health at 22:16 UTC. Residual issues with code search cleared at 01:11 UTC on October 12.
After public services were restored, engineers continued rebuilding original functionality within the affected site. GitHub says it is working on hardening resiliency and automation processes around this infrastructure to speed up diagnosis and resolution in future incidents.
