March 2023: Six Incidents Degrade GitHub Services
GitHub experienced six incidents in March 2023 that caused degraded performance across its services. The company has also disclosed a February incident that affected GitHub Codespaces. Below is a breakdown of each event and the steps being taken to prevent recurrence.
February 28: Codespaces Creation Failures in East US
On February 28 at 15:42 UTC, monitors detected a higher than normal failure rate when creating and starting Codespaces in the East US region. The cause was slower than normal VM allocation time from GitHub's cloud provider. To reduce customer impact, codespace creations were redirected to a secondary region; Codespaces in other regions were unaffected.
In response, GitHub tuned its monitors for quicker detection and is working on architectural changes to enable automatic failover for existing codespaces without human intervention.
March 1: Database Disk Issue Hits Package Registries
At 12:01 UTC on March 1, monitors flagged higher than normal latency for Container, NPM, NuGet, and RubyGems Packages registries. Incident impact began at 0.5% of requests returning 5xx errors and peaked at 10%. The root cause was an unhealthy disk on a VM node hosting MySQL, causing OS-level performance degradation. All MySQL servers on that node experienced connection delays and slow query execution.
GitHub mitigated the issue via database failover but acknowledged the incident lasted too long. Updated runbooks enable quicker mitigation, and architectural changes are in progress to migrate the application backend to a new MySQL infrastructure with improved observability and auto-recovery tooling.
March 2: CDN Certificate Binding Fault Disrupts Actions
At 23:37 UTC on March 2, an elevated number of GitHub Actions workflow failures appeared, caused by TLS verification failures linked to an unexpected SSL certificate bound to a CDN IP address. The CDN provider's configuration change unintentionally altered SSL certificate bindings for some Actions production IP addresses. The issue was remediated by removing the certificate binding.
GitHub is evaluating multiple DNS/CDN providers to maintain consistent networking even when issues arise outside its control.
March 15: Maintenance Query Blocks Package Registries
On March 15 at 14:07 UTC, increased latency was seen in Container, NPM, NuGet, and RubyGems registries. The cause: a slow running query that was not properly drained during routine maintenance on the primary database host blocked all database resources. The fix involved killing the query and restarting the database. Maintenance has been paused until long-running query draining is addressed, and the maintenance process now includes extra safety checks for long-running queries and blocking processes.
March 27: pages, Codespaces, and Issues Impacted
GitHub was notified of impact to pages, codespaces, and issues at 12:25 UTC on March 27, with resolution at 13:29 UTC. Due to the incident's recency, contributing factors are still under investigation, with a detailed update expected in the next monthly report.
March 29 and March 31: Two Further Incidents Under Review
At 14:21 UTC on March 29, an incident affecting pages, codespaces, and actions lasted 4 hours and 29 minutes, ending at 18:50 UTC. On March 31 at 01:16 UTC, Git operations, issues, and pull requests were impacted for 52 minutes, with resolution at 02:08 UTC. Both incidents remain under investigation, and detailed analyses will follow in next month's report.
For real-time status updates, follow GitHub's status page. More on engineering efforts is available on the GitHub Engineering Blog.
