Actions, API, and Pages All Affected in GitHub’s March Incidents
GitHub’s availability report for March 2021 details three separate incidents that contributed to degraded performance across issues, pull requests, webhooks, API requests, GitHub Pages, and GitHub Actions. A months-old database migration strategy and an external authentication outage were among the root causes.
Check Suite ID Overflow Hits Actions Jobs and Search
The first incident, which began March 1 at 09:59 UTC and lasted one hour and 42 minutes, was a continuation of a problem first reported in February. A high error rate on creating check suites for workflow runs caused some queued Actions jobs to fail or be delayed, and incomplete search results for workflow runs in both the UI and API.
The root cause traced back to check suite IDs exceeding the maximum Int32 value. GitHub had anticipated this limit and migrated all database columns to bigint six months earlier. However, while their own Ruby, Go, and C# code did not rely on explicit Int32 type casting, an external GraphQL library used Int32 when unmarshalling JSON, which caused the processing of successfully created check suites to fail. Jobs were never queued, leaving check suites in a pending state. A code fix was deployed after validating it would not introduce data integrity issues in other microservices dependent on check suite IDs.
The same ID overflow also impacted the data indexing for workflow run search. The index had to be rebuilt, during which search results remained incomplete. In response, GitHub has audited all external library usage and is using drill tests that simulate similar failures, such as check run IDs exceeding max Int32, to prevent recurrence.
Index Flip Leads to Table Scan and Timeouts
A second incident on March 12 at 19:11 UTC lasted one hour and 10 minutes. A database migration designed to reverse the order of an index for better query performance instead caused a full table scan. A generated ActiveRecord query still depended on the old index order, and the resulting performance degradation cascaded into query timeouts across dependent services.
GitHub is now looking for better tooling to identify index regressions and has created an inventory of indexes used by generated queries to ensure compliance with ActiveRecord best practices.
Authentication Provider Outage Halts Hosted Builds
The third incident, on March 15 at 20:38 UTC, lasted one hour and 18 minutes and impacted GitHub’s hosted Actions runners. The service’s monitors detected high failure rates and an inability to run hosted jobs. All hosted jobs were queued for extended periods before being abandoned because virtual environments could not be served. Build requests delayed by more than 30 minutes went unprocessed entirely.
The cause was an outage at the authentication provider used for the infrastructure that runs hosted compute. With the provider unavailable, no new hosted builds could be started. Once the authentication service recovered, GitHub processed the backlog quickly. To reduce reliance on this mechanism in the future, the company is exploring extending the lifetime of authentication tokens to better withstand brief outages.
