January Service Disruptions: Three Incidents, Three Different Root Causes

GitHub recorded three incidents in January 2025 that degraded performance across its services. Each outage stemmed from a distinct failure mode: a database query issue, a traffic routing misconfiguration, and a hardware failure in the caching layer.

Database Query Saturation (January 9)

Between 01:26 and 01:56 UTC, many GitHub services returned 500 errors to users. A deployment introduced a query that saturated a primary database server. The average error rate was 6%, with update requests peaking at 6.85%.

The mitigation involved identifying the problematic query and rolling back the deployment. Internal tooling and dashboards surfaced the data needed to locate the query within 14 minutes of engagement. GitHub is investing in tooling to detect such queries before deployment, aiming to shorten both detection and mitigation time for similar issues.

Git Operations Outage (January 13)

All Git operations were unavailable from 23:35 UTC until 00:24 UTC. A configuration change related to traffic routing and testing caused the internal load balancer to drop requests between services that Git depends on. Rolling back that configuration change resolved the incident.

GitHub is improving its monitoring and deployment practices to speed up detection and enable automated mitigation for configuration-related disruptions in the future.

Caching Layer Hardware Failure (January 30)

From 14:22 to 14:48 UTC, web requests to github.com failed with an error rate peaking at 44%. Successful requests took over three seconds on average. The cause was a hardware failure in the caching layer that supports rate limiting. The impact was prolonged because the caching layer lacked automated failover; a manual failover of the primary to trusted hardware was performed after recovery to prevent recurrence under similar conditions.

To address this, GitHub plans to adopt a high availability cache configuration and add resilience to cache failures at that layer, ensuring requests can still be handled if similar hardware failures occur.

For real-time updates on service status and post-incident recaps, see the status page. More details on ongoing engineering work are available on the GitHub Engineering Blog.