May 2022: Three Incidents, Two Investigations Open

GitHub experienced three availability incidents during May 2022, two of which remain under investigation. The company has also published a detailed post-mortem for an April incident that incorrectly billed GitHub Actions and Codespaces customers.

Container Registry Database Saturation

At 09:44 UTC on May 20, monitoring detected elevated CPU usage on one of the GitHub Container registry databases. On-call engineers, already primed by monitoring added after a similar April incident, began investigating immediately.

The rising CPU load traced to a high volume of Put Manifest commands. As CPU utilization climbed, the Container registry responded with increased latency, then began returning internal server errors for a percentage of requests. The root cause: API-side throttling criteria for this command were too permissive, and an underlying database query proved non-performant at that scale. Other package registries were unaffected.

Users pushing or pulling packages experienced latency, as did those accessing the packages UI. To mitigate, GitHub throttled requests from all organizations and users, then restored normal operation by restarting front-end servers followed by the database.

Preventative measures include separate rate limiting for specific operation types from individual organizations/users, along with ongoing SQL query performance work. The incident lasted 49 minutes.

On May 27, two additional incidents occurred. The first began at 04:26 UTC and lasted 21 minutes, with alerting detecting degraded availability for API requests. The second started at 07:36 UTC and persisted for 1 hour 21 minutes, impacting GitHub Actions, API Requests, Codespaces, Git Operations, Issues, GitHub Packages, GitHub Pages, Pull Requests, and Webhooks.

GitHub states it is still investigating contributing factors for both incidents. Detailed updates on causes and remediations are expected in the June Availability Report.

April Billing Incident Follow-Up

GitHub has provided a more detailed analysis of the April 14 incident that lasted 4 hours and 53 minutes, in which Actions and Codespaces customers saw incorrect charges for metered services in their billing settings. Affected customers began hitting spending limits and were unable to run new Actions jobs or create new Codespaces.

The incident bridge immediately unblocked all customers by providing unlimited Actions and Codespaces usage at no charge during the outage. Through timing and recent change analysis, engineers traced the problem to a code change in the metered billing pipeline.

A performance improvement to the metered usage processor mistakenly multiplied Actions and Codespaces minutes by 1,000,000,000 — a conversion meant to turn gigabytes into bytes — which was unnecessary for these products. The change was made to shared metered billing code and was not expected to affect these products.

The fix involved reverting the code change and repairing corrupted billing data recorded during the incident. Metered billing for GitHub products remained disabled until data repair completed, about 24 hours after the incident began.

To prevent recurrence, GitHub has added a Rubocop rule to block pull requests containing non-safe billing code updates, implemented anomaly monitoring on billed quantities to prompt earlier alerts, and tightened the release process to require a feature flag and end-to-end tests for such changes.

Status Updates

GitHub continues to provide real-time status updates via its public status page and publishes reliability engineering details on its Engineering Blog.