Codespaces and Packages outages hit GitHub in April
GitHub’s April availability report describes three incidents that produced significant degradation for Codespaces and GitHub Packages. Two of the three required several hours each to resolve.
Expired secrets drain Codespaces capacity
The longest incident began April 1 at 7:07 UTC and lasted five hours and 32 minutes. Alerting picked up an increase in failures to create new Codespaces and to start existing stopped ones in the US West region. The GitHub status page was updated immediately while engineers investigated.
The root cause turned out to be expired secrets used by the Codespaces service. Codespaces keeps warm pools of resources to buffer users against intermittent dependent-service failures, but in US West those pools were empty because of the expired secret. There was no early warning that pool levels were getting low, so the team had no chance to react before capacity ran out. While mitigation was underway, pools in other regions also drained due to the same expired secret, and those regions started seeing failures as well.
Only a limited number of GitHub engineers had permission to rotate the secret, and communication problems delayed the start of the refresh. Once the secret was refreshed and rolled out across all regions, the service returned to normal operation.
To avoid this pattern recurring, GitHub now verifies resources that expire and has monitors in place that alert well in advance when pool resources are not being maintained. Additional monitors now also fire earlier when the service approaches resource exhaustion. The team has also started migrating the service to a mechanism that does not depend on secrets or require credential rotation.
Packages database CPU spike takes down registry
On April 25, starting at 8:59 UTC, an incident lasting five hours and 8 minutes affected anyone using the GitHub Packages Registry. Alerting detected increased CPU utilization on one of the Packages Registry databases about one hour before any customer impact. However, the alert threshold was relatively low and was not a paging alert, so no immediate investigation occurred. CPU continued to climb, and the Package Registry began responding to requests with internal server errors, eventually causing customer impact.
The increased activity was traced to a high volume of the “Create Manifest” command being used in an unexpected manner. The throttling criteria configured at the database level was insufficient to limit the command, and the result was an outage for the Packages Registry: users could not push or pull packages, and they could not access the packages UI or the repository landing page.
The investigation revealed a performance bug related to the high volume of “Create Manifest” commands. To restore normal operation and limit impact, GitHub blocked the activity causing the problem. The team is following up by improving rate limiting in packages and fixing the underlying performance issue. Database alerting thresholds and severity have also been modified so unexpected issues trigger alerts more quickly, before customer impact occurs.
The incident also exposed a hard dependency that the repository home page has on packages infrastructure. When the package registry is down, home pages for repositories that list packages fail to load as well. During the outage, decoupling the package listing from the repository home page required manual intervention. GitHub is working on a fix that loosely binds the packages listing so that a failure there does not bring down repository home pages for repositories that list packages.
April 14 incident still under investigation
GitHub reported a third incident on April 14 at 20:35 UTC, lasting four hours and 53 minutes. Contributing factors are still being investigated, with a more detailed update planned for the May Availability Report, published the first Wednesday of June.
