December Sees Three Availability Incidents Tied to Secret Rotation
GitHub experienced three separate incidents in December that each resulted in degraded availability across various services. All three trace back to a company-wide secret rotation initiative that took place in late December. While each incident has been individually reviewed and remediated, GitHub says it is also examining broader ways to reduce availability risk within its secrets management architecture.
Codespace Creations Fail During Key Rollback
The first incident began on December 27 at 02:30 UTC and lasted 90 minutes. During an HMAC secret rotation between GitHub's frontend service and an internal service, a bug was triggered in the process used to fetch keys from Azure Key Vault. When a key was disabled in Key Vault during a rollback—initiated in response to an alert—API calls between the two services began failing.
The impact caused all codespace creations to fail between 02:30 and 04:00 UTC. Roughly 15% of codespace resumes failed as well, along with other background functions. To mitigate the issue, the team temporarily re-enabled the key in Key Vault before deploying a change that allowed the secret rotation to proceed. The original alert turned out to be unrelated to customer impact and was resolved immediately after the incident ended.
Post-incident, the team has refined playbooks for HMAC key rotation and expanded documentation of GitHub's Azure Key Vault implementation.
Email Notifications Interrupted by SMTP Authentication Miss
The second incident spanned 65 minutes on December 28, from 05:52 to 06:47 UTC. During that window, certain GitHub email notifications were not delivered. The failure was caused by an authentication breakdown: credentials used by backend services generating notifications did not align with a subset of GitHub's SMTP servers.
The issue primarily affected CI activity and Gist email notifications. It originated from a credential rotation between frontend and internal services that did not propagate correctly to the affected SMTP servers. An alert for one of the two impacted notifications services fired within minutes of the secret rotation, enabling on-call engineers to detect the misconfigured authentication and correct it to mitigate the outage.
Repair items for this incident include updates to the relevant secrets rotation playbooks and documentation. Although the alert that fired did succeed in engaging the on-call team, an additional repair item has been completed to extend earlier alerting to all services going forward.
Sign-In and Sign-Up Failures on December 29
The third incident occurred on December 29, lasting 68 minutes from 00:34 to 01:42 UTC. During this window, users could not sign in to GitHub or create new accounts. Existing sessions remained unaffected.
The root cause was a credential rotation that was not mirrored in GitHub's frontend caches. This created a discrepancy in behavior between authenticated and unauthenticated users. The incident was resolved once the updated credentials were deployed to the cache service.
Repair items are still underway for this incident. GitHub is working on improving monitoring of signed-out user experiences and developing a better process for managing updates to shared credentials across its systems.
