November’s Availability Recap: Two New Incidents, One October Follow-Up
GitHub’s November availability report covers two incidents that caused degraded performance during the month, plus a detailed retrospective on an October outage that affected Codespaces users in the East US region.
November 25: Packages and Pages Degradation
At 16:34 UTC on November 25, monitoring systems flagged a problem impacting GitHub Packages and Pages. The incident lasted 1 hour and 56 minutes. GitHub notes that the investigation is still ongoing, and the company plans to publish a full analysis of root cause and remediation steps in the January Availability Report, due out on the first Wednesday of January.
October 26: Codespaces Outage in East US
Although this incident occurred in late October, it is detailed in the November report. It began at 00:47 UTC on October 26, when alerting detected a decline in success rates for creating and resuming Codespaces in the East US region. GitHub initially posted a yellow status since impact was limited to that region. At 01:55 UTC, affected customers were advised to manually switch their location to a nearby healthy region. The status was elevated to red at 2:34 UTC as the outage stretched on.
During the incident, users in East US could not create or resume Codespaces. While they could manually select an alternate region for new codespaces, the same workaround was not available for resuming existing ones.
The root cause lay with a third-party database provider that GitHub Codespaces relies on for data storage. The provider experienced its own outage, and GitHub’s service was unable to communicate with its East US database because regional data is not currently replicated. The service returned to healthy once GitHub engineers worked with the third party to resolve the provider-side issue.
As a result of this post-incident review, GitHub is implementing regional replication and failover capabilities to mitigate similar regional database failures more quickly in the future.
November 3: Webhook Backlog Delays GitHub Actions
A shorter but broader incident occurred on November 3. At 16:10 UTC, alerting picked up an increase in the time it took for GitHub Actions workflow runs to start. GitHub initially set the status for Actions to red, then downgraded to yellow at 16:11 UTC after assessing the scope.
Users saw high latency on webhook deliveries, delays starting workflow runs, and slow status updates for jobs already in progress. Error rates also rose across repositories, pull requests, Codespaces, and the GitHub API. At the peak, most repositories attempting to launch a workflow experienced delays exceeding five minutes.
The investigation traced the problem to a backup in the webhooks queue, which GitHub Actions depends on to trigger runs. At 16:29 UTC, engineers scaled out and accelerated queue processing. The queue was fully drained by 17:12 UTC, and the status returned to green.
The underlying cause was an inefficient database query used to check repository security advisory access. A high volume of poorly optimized API calls triggered this query, which in turn backed up background jobs across GitHub and caused the widespread impact. After the incident, GitHub fixed the inefficient query and revisited the rate limits on the endpoint in question to align them with expected performance. Default throttling for the webhooks queue was also increased to reduce the chance of future backups. Longer term, GitHub is exploring ways to limit the impact of other background jobs on Actions workflows, and will continue running game days and refining first-responder training for better incident assessment and statusing.
