August 2025: Three Incidents Cause Service Degradation on GitHub
GitHub experienced three separate incidents during August 2025 that led to degraded performance across multiple services. Each incident was traced to distinct root causes, though two shared a common trigger involving database migrations and ORM behavior.
Database Migration Error Hits Pull Request Services (August 5)
At 15:33 UTC on August 5, a production database migration was initiated to remove an unused column from a table backing pull request features. Although the column was no longer in active use, the application's ORM still referenced it in certain pull request queries. This created elevated error rates across pushes, webhooks, notifications, and pull requests, with impact peaking at roughly 4% of all web and REST API traffic.
The issue was mitigated by deploying a change that made the ORM ignore the removed column, with most services recovering by 16:13 UTC. However, the fix was only applied to the largest production environment. Custom and canary environments missed the update, causing a second, smaller incident that affected ~0.1% of pull request traffic until it was fully resolved by 19:45 UTC.
The incident exposed a gap in application monitoring. While migrations already include protections like progressive rollouts targeting validation environments and acknowledge gates, the rollout continued despite visible impact. GitHub plans to add more automation and safeguards that can stop similar issues without human intervention, and is working on streamlining certain changes across environments to prevent recurrence of the secondary incident.
Search Outage Overwhelms Load Balancers (August 12)
Between 13:30 UTC and 17:14 UTC on August 12, GitHub search was degraded. Users saw inaccurate or incomplete results, and some pages—including issues, pull requests, projects, and deployments—would not load. Components such as actions workflow and label filters also broke.
The most severe impact came between 14:00 UTC and 15:30 UTC, when up to 75% of search queries failed and search result updates were delayed by as much as 100 minutes. Intermittent connectivity issues between load balancers and search hosts triggered the incident. Retry logic initially masked the problems, but the retry queues eventually overwhelmed the load balancers, causing failures. The query failures were mitigated at 15:30 UTC by throttling the search indexing pipeline to reduce load and stabilize retries. After an automated reboot of a search host, the connectivity issues resolved at 17:14 UTC and the remainder of the system recovered.
Post-incident work includes improved internal monitors, updated playbooks, and adjustments to the search cluster load balancer configuration. A configuration issue in the load balancing tier was identified and fixed as the underlying trigger.
Copilot and Web Traffic Degraded After Similar Migration Issue (August 27)
From 20:35 to 21:17 UTC on August 27, Copilot, web, and REST API services performed poorly. Copilot saw an average of 36% of requests fail, with a peak failure rate of 77%. Around 2% of all non-Copilot web and REST API traffic also failed.
The root cause mirrored the August 5 incident: a production database migration dropped a column from a table backing Copilot functionality, and the ORM still referenced the removed column, producing a wave of 5xx responses. A fix was applied to the production schema at 21:15 UTC, and all services had recovered by 21:17 UTC.
Although repairs were in progress to prevent this situation, they were not completed fast enough to stop the second occurrence. As an immediate measure, GitHub has temporarily blocked all drop column operations while adding permanent safeguards. Work is also underway on graceful degradation so that Copilot issues do not affect other parts of the product.
Real-time status updates and post-incident recaps are available on the GitHub status page. Engineering details on ongoing work can be found on the GitHub Engineering Blog.
