March 2025: One Degradation Event, Two Root Causes
GitHub reported a single incident of service degradation during March 2025. The disruption stemmed from two related credential-expiry failures that affected customer-facing features over a 58-hour window.
Timeline and Impact
The degradation began on March 29 at 7:00 UTC and lasted until March 31 at 17:00 UTC. On the first day, users could not unsubscribe from GitHub marketing email subscriptions through the github.com/settings/emails interface, nor could they complete the double opt-in step required in some countries. This was caused by expired credentials for an internal service.
A separate but similar credential expiry on March 31 (from 7:00 UTC to 16:40 UTC) prevented users from submitting ebook and event registration forms on resources.github.com.
Root Cause and Mitigation
Investigators traced the issue to a bug in the automated alerting system designed to monitor upcoming credential expirations. The defect meant invalid credentials were not detected until after they had already lapsed, leading to two distinct incidents before a durable fix could be deployed.
GitHub mitigated the problem by renewing the affected credentials and redeploying the impacted services.
Preventive Measures
To reduce the chance of recurrence and improve response times, GitHub has enhanced its credential expiry detection, alerting, and rotation processes. The company is also working on improving on-call observability.
For real-time status updates and post-incident reports, follow the GitHub status page. Engineering details are available on the GitHub Engineering Blog.
