June 2024 Availability Report: Two Service Disruptions
GitHub experienced two separate incidents in June 2024, each causing degraded performance in distinct areas of the platform. The first affected the Issues service, while the second impacted the Migration service.
Issues Service Degradation: June 5
Between 17:05 UTC and 19:27 UTC, GitHub Issues was degraded, with project event data failing to appear on issue timelines. These events would normally display when an issue was added to or removed from a project, along with any associated status changes within that project.
An internal cleanup initiative aimed at simplifying service configurations for better automation led to the incident. A bug in that implementation caused a scheduled secret rotation to leave one of the underlying services using old, expired secrets. The resulting misconfiguration prevented the service from loading the event data.
The team mitigated the issue by correcting the service configuration. As a result of the cleanup effort, the simplified configuration is expected to reduce the likelihood of similar problems emerging in the future.
Migration Service Outage: June 27
From 20:39 UTC to 21:37 UTC, the GitHub Migration service experienced failures across all in-progress migrations. After detecting the increase in failures, the team paused new migrations, which extended overall migration times but prevented additional failures from occurring.
The root cause was traced to an invalid infrastructure credential requiring manual intervention. Once identified, first responders resolved the issue, and queued migrations were unpaused and processed normally.
To reduce the risk of recurrence, GitHub is addressing specific gaps identified in the monitoring and alerting systems that cover infrastructure credentials.
