GitHub Reports Two March Incidents Affecting Core Services
GitHub has published its availability report for March 2024, detailing two separate incidents that led to degraded performance across multiple services. Both issues were resolved, though each required distinct remediation steps and follow-up measures.
The first and shorter incident occurred on March 15, lasting 42 minutes from 19:42 to 20:24 UTC. A regression in the permissions system caused failures in GitHub Codespaces, GitHub Actions, and GitHub Pages. The root cause was traced to a framework upgrade that introduced MySQL query syntax incompatible with the database proxy service used in select production clusters. GitHub rolled back the deployment and corrected a misconfiguration in development and CI environments to prevent recurrence.
The more significant outage preceded it on March 11, lasting 2 hours and 3 minutes from 22:45 to 00:48 UTC. An inadvertent deployment of network configuration to the wrong environment produced intermittent errors across API requests, GitHub Copilot, GitHub secret scanning, and 2FA via GitHub Mobile.
The issue was detected within 4 minutes, and a rollback began immediately, mitigating the majority of impact by 22:54 UTC. However, the rollback failed in one data center because system-created configuration records lacked a required field, leaving 0.4% of requests failing. Full restoration only succeeded after manual intervention corrected the configuration data, with services fully operational by 00:48 UTC.
In response to the March 11 incident, GitHub has implemented safeguards for future configuration changes, including prevention mechanisms for invalid records, automatic cleanup of obsolete configuration, and faster detection of similar issues.
For real-time updates and post-incident reports, GitHub directs users to its status page. Further engineering details are available on the GitHub Engineering Blog.
