Monthly Availability Reporting
GitHub is introducing a monthly Availability Report, published on the first Wednesday of each month. The report covers service availability for the preceding period, describes any incidents that affected GitHub.com, and outlines the engineering changes made in response. The goal is to move beyond bare status-page metrics and share the technical context behind incidents so that other engineering teams can benefit from GitHub’s experience.
Designing for Transparency
GitHub already publishes post-incident reviews for major outages, but those are reserved for the most significant events. The new monthly report is intended to cover all incidents that may have impacted users, including shorter or less severe disruptions. For incidents with novel failure modes or broadly applicable lessons, the report will include detailed technical explanations. Months where availability remained above 99% will be recapped as such, with the emphasis on what was learned rather than just reporting minutes of downtime.
May and June Incidents
May 5: Database ID Column Exhaustion
An auto-incrementing ID column in a shared database table exceeded the maximum value representable by the MySQL Integer type (Rails int(11)). Inserts with values larger than 2147483647 were rejected by the database, causing Rails to raise an ActiveModel::RangeError and the API to return 500 errors.
The failure affected GitHub apps that depend on installation tokens, with Actions, Pages, and Dependabot among the most impacted internal consumers. Monitoring systems currently alert at 70% of primary-key capacity; test frameworks are being extended with a linter to catch int/bigint foreign key mismatches before they reach production.
May 22: Novel Crash During Planned Failover
While performing a planned MySQL primary failover, the newly promoted primary experienced a crash in the mysqld process. Because the crashed host had already served roughly six seconds of write traffic, GitHub redirected traffic back to the original primary. Restoring replicas from the new primary took approximately four hours, followed by another hour of cluster reconfiguration to restore full read capacity. For about five hours, users may have seen delays before writes to the affected cluster appeared in the web UI and API.
GitHub has run multiple internal gameday exercises since the incident to improve preparedness for similar topology inconsistencies and continues to test its automated failover systems to shorten recovery times.
June 19: A/B Experiment Dependency Failure
New instrumentation for A/B experimentation introduced an implicit dependency on a dynamically generated file served by a separate application. During a deployment, that file failed to be generated on a significant number of application instances because a high retrieval rate was rate-limited by the upstream service. A subset of users enrolled in the experiment encountered site-wide application errors. Once detected, GitHub disabled the file requirement and restored service.
Configuration for A/B and multivariate experiments is now cached internally to ensure dependency propagation is reliable.
June 29: ProxySQL Update and Repeated Primary Crashes
The database team had deployed an updated version of ProxySQL on June 22 as part of routine maintenance. A week later, the primary MySQL node on a main database cluster failed and was replaced automatically. Within seconds, the new primary crashed. Organizer’s anti-flapping mechanism then prevented another automatic failover. After manual recovery, the replacement primary became CPU-starved and crashed again; a second promoted primary also crashed shortly after promotion.
Recovery involved rolling back to the prior ProxySQL version and disabling the application change that had required the newer version. Writes were then re-enabled on the primary without further crashes. GitHub is still analyzing application logs, MySQL core dumps, and internal telemetry to understand the CPU starvation issue and prevent similar failure modes.
What to Expect Going Forward
GitHub views each incident as an opportunity to improve both its systems and its operational practices. The availability report is part of a broader commitment to reliability engineering and accountability. Real-time updates remain available on the status page, while the monthly report will be posted on the blog. Follow along at Organizer for deeper technical context on some of the database tooling mentioned in incident details.
