Two Availability Incidents Disrupt GitHub Services in August
GitHub experienced two separate incidents during August 2021, each with distinct root causes and service impacts. The first, on August 10, degraded core Git operations and write-dependent features for over an hour. The second, later that same day, caused a multi-hour failure of GitHub Actions workflow jobs.
Database Primary Degradation Halts Write Operations
The first incident began at 15:16 UTC on August 10 and lasted one hour and 17 minutes. A MySQL database primary entered a degraded state, interrupting write access for services dependent on that specific cluster. Users encountered failures when performing Git operations, API requests, or other actions requiring database writes.
Investigation traced the cause to an edge case in one of GitHub's most active applications. The flaw produced a poorly performing query that consumed excess database capacity. Combined with existing application retry and queueing logic, this pushed the MySQL primary into a state from which the cluster could not automatically recover.
GitHub has since addressed the problematic query and adjusted portions of the retry logic to reduce the probability of recurrence. The incident also triggered internal review of how service status is reported during multi-service failures, with continued tuning of metrics and alerting systems to improve accuracy and speed of status communication.
Service Discovery Failure Takes Down Actions
The second incident began at 19:57 UTC on August 10 and lasted 3 hours and 6 minutes, affecting GitHub Actions only. This was unrelated to the earlier database issue. Service monitors detected a high error rate on both new and in-progress workflow runs shortly after ongoing maintenance on the Actions service, resulting in failure of all queued jobs for a period of time.
GitHub immediately reverted recent Actions deployments and began investigation. The root cause was traced to work establishing a new Actions Premium Runner microservice. Specifically, changes to service discovery within the Actions microservices architecture introduced a bad service record. This record prevented many microservices from making Service-to-Service calls.
The fix involved removing the bad record from the service discovery infrastructure. After confirming this would not produce unintended side effects, GitHub applied the mitigation and restored Actions service functionality.
As follow-up, GitHub has prioritized fixes to the Actions microservice discovery process so it properly handles potential bad records. The team also expanded visibility into recent changes across all Actions microservices to speed up future investigations.
For real-time updates on service status, follow the GitHub status page. Further engineering details are available on the GitHub Engineering blog.
