GitHub Availability Report: August 2024
GitHub experienced one service-degrading incident in August 2024. The outage affected all services on GitHub.com for a 36-minute window on August 14.
Global Service Outage on August 14
Between 23:02 and 23:38 UTC, users were unable to access any GitHub.com service. The root cause traced back three minutes earlier, at 22:59 UTC, when an erroneous configuration change was rolled out to GitHub's databases. That change interfered with the databases' ability to respond to health check pings from the routing service.
Because the database hosts failed those health checks, they were marked as unhealthy, and the production read-only database endpoint became unreachable. The application consequently lost the ability to connect to critical data for read operations, which triggered the widespread impact beginning at 23:02 UTC. No data loss or corruption occurred during the incident.
Engineers mitigated the problem by reverting the configuration change and verifying that connectivity to the databases was restored. Traffic resumed at 23:38 UTC, and all services returned to full health. As a precaution, monitoring continued before the incident was resolved at 00:30 UTC on August 15.
Preventive Measures
To avoid a repeat of this failure, GitHub has added guardrails to its database change management process. The company is also prioritizing repair items including faster rollback functionality and improved resilience to dependency failures. Given the severity of the incident, all repair work is being handled at the highest priority level.
