Codespaces Configuration Change Caused April’s First Outage
GitHub’s April availability report covers three incidents that caused degraded performance. The first, on April 11, hit Codespaces users especially hard.
From 03:05 UTC to 03:44 UTC, approximately 75% of Codespaces users encountered create and start failures. The root cause was a manual configuration change to an internal dependency that slipped past test coverage. Monitors and detection mechanisms fired, enabling triage and a quick revert to restore service health.
GitHub says it is building additional gates, safer testing mechanisms, and safer rollout procedures for such configuration changes, and expects no further disruptions.
Database Contention Caused Mid-April Degradation
Later in the month, on April 23, a broader set of services saw problems between 07:00 UTC and 07:20 UTC. Resource contention on database hosts resulted in error rates of 2–5% of total requests, causing intermittent disruption.
The trigger was an interaction between query load and an ongoing schema change that led to connection saturation. The incident resolved once the schema migration completed. GitHub credits prior investments in monitoring and improved playbooks for helping first responders triage quickly. A regression in the schema change tooling was identified and rolled back to a previous stable version.
Future preventive steps include reviewing database capacity, improving monitoring and alerting, and adding safeguards to shorten time to detection and mitigation.
Migration Service Workers Lost Access in Third Incident
Later that same day, GitHub’s Migration service saw elevated failures. Between 19:13:50 UTC and 22:11:00 UTC, a configuration change removed access for repository migration workers. During the window, 837 migrations across 57 organizations were affected.
Impacted users saw the log message “Git source migration failed. Error message: An error occurred. Please contact support for further assistance.” and had to retry once the team restored access. Normal operations resumed without further interruption, and GitHub says it has since improved test coverage and refined monitoring thresholds.
