October at a glance
Four incidents in October caused significant impact across several GitHub services, with Codespaces bearing the brunt of the disruption. GitHub also used this report to detail a September incident involving Codespaces secret rotation that surfaced only later. The most recent October incident, on October 26, remains under investigation and will be detailed in the November availability report.
Projects API error spike after a bad deployment
On October 13 at 20:43 UTC, alerting detected an increase in the Projects API error rate. Within four minutes, the impact was deemed significant enough to move Issues to red status. The root cause was traced to a recently deployed change: a database validation that required a specific value to be present but did not set a default in every code path. This led to null values being written, which then caused errors when records were retrieved.
Engineers rolled back the change at 21:08 UTC. Error responses began falling by 21:13 UTC, Issues returned to yellow at 21:24 UTC, and went back to green at 21:31 UTC once all metrics were healthy.
Post-incident work included adding protections against missing values, improving test coverage in that area, and fixing deployment dashboards that had shown inaccurate pre-production error data. The dashboard fix is meant to make deployment-time errors more visible before they reach production.
Webhooks backlog triggered by automated repository churn
On October 5 at 06:30 UTC, a large volume of automated user activity—rapid create and delete operations on repositories—generated a flood of webhook events. The problem: many of these events caused exceptions in webhook delivery workers because the underlying data had been deleted from the database. Failed retries consumed worker capacity, and new events piled up in queues. Downstream services could not receive webhook events, and GitHub Actions was marked red because new job execution was severely delayed.
The source was automation creating and deleting many repositories in quick succession. GitHub disabled the offending automated accounts as a mitigation, which allowed webhook deliveries to return to normal and cleared the backlog by 07:01 UTC. Webhook delivery workers were also updated to skip retries for jobs where the required data no longer exists in the database. Accounts were re-enabled after the fix, with no further issues.
Codespaces sidelined by a global configuration change
On October 12 at 22:30 UTC, a global configuration change was rolled out for Codespaces. By 23:15 UTC, new Codespace creation was trending downward across regions, and the service was statused yellow at 23:27 UTC before moving to red as impact widened.
The investigation found that an older backend component did not handle the configuration change well, triggering a schema conflict. That component had not been properly tested before rollout and does not support gradual regional exposure, so numerous regions were affected at once. Rolling back the large schema change was complex and took considerable time. Once complete, new Codespace creation metrics recovered and the service returned to green at 02:58 UTC.
GitHub says it is working to eliminate the dependency on this older configuration type entirely and has verified that all future component changes will follow safe deployment practices, with one test region followed by individual region rollouts.
Missed secret rotation step hits Codespaces twice
On September 27 at 23:14 UTC, a routine secret rotation was performed on Codespaces. Early the next morning, at 03:21 UTC, an internal report indicated port forwarding was not working in the Codespaces web client. The service was statused yellow at 03:53 UTC. Investigation revealed that a step in the rotation checklist had been skipped hours earlier, causing some downstream components to fail to pick up the new secret and dropping traffic to backend services.
The missed step was run at 04:29 UTC, and port forwarding quickly returned to health. GitHub then noted that its alerts monitored error rates but not overall traffic volumes to the port forwarding backend, so no automated alert fired. Monitoring has since been expanded to include traffic-level anomalies.
The same root cause surfaced again later that day. At 17:18 UTC, monitors flagged issues in a separate downstream component—Codespaces creation and start failures rose in all regions. The delay occurred because the affected secret is used in exchange for a token, which is cached for up to 24 hours. Engineers had expected components to pick up the new secret without intervention, but only restarted processes actually did so. Restarting the service across all regions at 18:27 UTC helped VM pools begin recovering, and pool sizes were increased at 18:45 UTC to clear the backlog of queued jobs. Most regions recovered, but two West Europe pools remained stuck until 19:44 UTC, when an instance that had not been rotated was identified and fixed.
Since then, GitHub has added monitoring to track which secret versions are in use across all service components, updated the rotation checklist with the missing steps and added verification checks, and is working to automate most secret rotation processes to reduce the risk of human error.
