September 2022 Availability Overview
GitHub experienced a service-degrading incident in September that affected multiple services, along with a separate incident that significantly impacted Codespaces. The Codespaces incident is still under investigation and will be detailed in next month's report. This report also includes retroactive details on August incidents that affected Codespaces and GitHub Actions.
Pull Request Merge Failures for Managed Users
On September 8 at 19:44 UTC, monitoring detected an increase in pull request merge failures. The impact was isolated to Enterprise Managed Users (EMUs), with a small number of bot accounts also affected.
Within 45 minutes, the root cause was traced to a data transition designed to remove inconsistent data from profile records. The transition incorrectly operated on EMU accounts, stripping data required for successful pull request merges via the UI and API. CLI merges were unaffected.
Data was restored from backup, though the process took longer than expected. A code-based workaround was considered but rejected due to the risk of introducing data inconsistencies. The restore resolved the issue, with pull request monitors recovering by September 9 at 00:55 UTC.
Post-incident improvements include revised data transition procedures that allow faster restores and automatic rollback without relying on backups. Testing processes related to EMUs are also being enhanced.
Codespaces Incident Under Investigation
An alert on September 28 at 03:53 UTC detected an incident impacting the majority of Codespaces customers. The incident lasted 1 hour and 16 minutes. Contributing factors are still being investigated, with a detailed update on cause and remediation to be published in the October Availability Report on the first Wednesday of November.
Retroactive: August 29 Codespaces DNS Failure
On August 29 at 12:51 UTC, monitoring detected an increase in Codespaces create and start errors, along with DNS-related networking failures in running Codespaces where outbound DNS resolutions were failing. The status was escalated from yellow to red at 14:19 UTC due to broad user impact.
The incident was caused by an Ubuntu security patch in systemd that broke DNS resolution. Recent Ubuntu versions enable unattended security upgrades by default, and Codespaces host VMs applied these patches automatically after VM creation. Once installed, the patch broke DNS resolution, leading to failures in creating or starting Codespaces and to outbound network call failures in existing Codespaces.
Once systemd's DNS resolver was identified as the source, the issue was mitigated by disabling it and manually configuring an upstream DNS resolver IP. The DNS configuration change was deployed at 18:13 UTC, positive recovery signals appeared at 18:21 UTC (status changed to yellow), and all metrics were fully healthy at 18:31 UTC.
Following this incident, DNS configuration is being updated to reduce dependence on systemd's DNS resolver. GitHub is also evaluating whether to continue using unattended upgrades; disabling them would provide more deterministic runtime behavior and prevent external changes from breaking Codespaces while preserving the ability to quickly patch VMs across the fleet.
Retroactive: August 18 GitHub Actions Token Service Degradation
This incident, which occurred on August 18 and lasted 3 hours and 23 minutes, was excluded from the August report because it did not constitute a widespread outage. It is being included retroactively due to the number of GitHub Actions customers affected.
At 14:13 UTC, a sudden traffic spike to GitHub Actions created an unusually high write load. Most services handled this gracefully, but an internal service used for generating security tokens began returning 503 Service Unavailable errors, triggering an alert. Investigation revealed that the token database was experiencing performance degradation, which, combined with the increased load, caused the database to hit its maximum concurrent connections limit. A mismatch between client-side throttling limits and database capacity compounded the issue by allowing more traffic than the database could handle.
Mitigation involved scaling up the impacted database and allowing a higher number of concurrent connections. The service returned to health and the incident was resolved at 17:36 UTC. Monitoring and alerting have since been improved for faster remediation, and throttling mechanisms are being reviewed to better handle similar traffic patterns.
