February’s Availability Snapshot
GitHub recorded three incidents in February that degraded service performance, and this report also details two earlier January events: one affecting GitHub Packages and Pages, and another impacting Git archive downloads.
Container Registry Authentication Failure (January 30, 21:31 UTC, 35 minutes)
At 21:36 UTC on January 30, monitoring flagged a surge of 500 errors on Container registry requests. This caused most GitHub Pages builds and GitHub Packages requests to fail.
The root cause was a configuration change deployed at 21:30 UTC to enforce Redis authentication. A hard-coded connection string in the production deployment file prevented clients from authenticating, breaking connections. GitHub reverted the Redis configuration change at 22:12 UTC; the registry began recovering two minutes later, and Pages was healthy by 22:21 UTC.
Going forward, GitHub has improved how secrets are managed in the registry’s Redis deployment configurations and added test coverage for authenticated Redis connections.
Source Code Archive Checksum Changes (January 30, 18:35 UTC, 7 hours)
A deployment at 18:35 UTC modified compression settings for source code downloads. While file contents were unchanged, the resulting checksums differed. Many communities relied on byte-for-byte stability, so this caused widespread disruption. After reverting the change, GitHub communicated with the affected communities.
The company acknowledges it underestimated the effect of this change and is implementing new procedures, including improvements to how Git is deployed across GitHub and the addition of checksum validation to its workflow. A related post outlines the plan for future stability of source code archives and hashes.
Southeast Asia Codespaces Outage (February 7, 21:30 UTC, 20 hours 35 minutes)
On February 7, failures creating, starting, and connecting to GitHub Codespaces in Southeast Asia were traced to a cloud provider datacenter outage. GitHub redirected new codespace creations to a secondary location to limit impact. Once the datacenter recovered, regional codespaces automatically restored and could be restarted, with no impact on other regions.
As a result, GitHub is evaluating expanded regional redundancy and has started architectural changes to handle temporary regional outages, including more frequent failover exercises.
Database Topology Corruption (February 18, 02:36 UTC, 2 hours 26 minutes)
At 02:36 UTC, application errors pointed to connectivity issues with MySQL databases, believed to arise from unhealthy deployments of sharding middleware. A re-deployment at 03:30 UTC inadvertently spread the issue to all Kubernetes pods, returning widespread 500 errors and blocking sign-ins to GitHub.com.
By 04:30 UTC, GitHub discovered corrupted database topology in 30% of deployments, which prevented applications from connecting. Applying the correct topology across all deployments resolved errors by 05:00 UTC, restoring sign-in capabilities.
Preventive measures include a new monitor for database topology errors and improved observability for topology reloads, both successful and failed. A deeper review is ongoing to address contributing factors.
Codespaces Performance Degradation (February 28, 16:05 UTC, 1 hour 26 minutes)
GitHub was notified of degraded Codespaces performance at 16:05 UTC on February 28, with the incident resolved at 17:31 UTC. Investigation is still ongoing, and a detailed update will appear in next month’s report.
