Actions service degraded for nearly four hours in late January
GitHub recorded one incident in January 2021 that caused significant impact to the Actions service. Between 04:21 and 08:14 UTC on January 28, some queued jobs failed or were delayed. Jobs queued during the outage were eventually executed successfully once the underlying problem was corrected.
What went wrong
The root cause was traced to an infrastructure failure in the SQL database layer. That failure affected a core microservice responsible for authentication and communication between the Actions microservices, disrupting queued jobs across the service. Under normal operation, automated processes would detect an unhealthy database and trigger a failover with minimal impact. In this case, the specific failure pattern was not recognized by the automation, and telemetry showed no database anomalies, which prolonged root cause analysis and mitigation.
Remediation steps
To prevent this class of failure from recurring, GitHub is updating the automation in its SQL database layer to improve error detection and failover handling. The company is also continuing its work on localizing failures to limit the blast radius of infrastructure errors. Further engineering updates are shared on the GitHub Engineering blog.
