GitHub Availability Report: November 2023
GitHub recorded a single service disruption in November, resulting in degraded performance across the platform for just over half an hour.
Authorization Outage on November 3
From 18:42 to 19:20 UTC on November 3, the GitHub authorization service consumed excessive application memory, which caused authorization requests to fail. As a result, users encountered 404 or error responses on most page and API requests.
The root cause was a memory leak introduced by a performance and resilience optimization applied to the authorization microservice. This leak only manifested under high traffic, but pre-production testing did not exercise the service at sufficient load levels to expose it, allowing the flawed build to reach production at 18:37 UTC. Once in production under real-world load, the leak caused pods to crash repeatedly beginning at 18:42 UTC. Since authorization checks default to a closed state, these crashes blocked requests across the platform.
Monitoring alerts triggered at 18:44 UTC, but the rollback process was slower than expected. Parts of the deployment infrastructure depended on the very authorization service that was failing, requiring manual intervention to complete the rollback. The fix was fully applied at 19:08 UTC, and all affected GitHub features recovered once pods resumed normal operation.
To prevent a similar recurrence, GitHub has updated its rollout strategy to include additional monitoring and health checks that automatically halt a deployment if key metrics degrade. The company has also decoupled the authorization service from the tooling used to perform rollbacks, cutting down the time needed to revert problematic changes in the future.
