Webhook Queue Backup Causes Widespread Delays
GitHub reported two incidents in August that caused degraded performance. The first and more severe event centered on webhook processing, while the second was a broader disruption affecting multiple background systems.
Webhook Deliveries Delayed by Load Spike
On August 15 at 16:58 UTC, GitHub observed growing delays in an internal job queue responsible for processing webhooks. The service was marked as degraded at 17:24 UTC. Customers saw delivery delays reaching 4.5 hours during the incident, which lasted 4 hours 29 minutes.
The root cause was a significant, sustained spike in webhook deliveries that backed up the delivery queue. Engineers brought the system back to normal by blocking events from the sources driving the increased load, then gradually worked through the backlog. In response, GitHub said it has made changes to handle higher traffic volumes and better absorb load spikes without increasing latency, and has strengthened its tools for managing load sources to speed up mitigation in the future.
Background Job Processing Disrupted on August 29
A second incident began on August 29 at 02:36 UTC and lasted 49 minutes, causing widespread delays in background job processing. Webhook deliveries, GitHub Actions, and other asynchronously-triggered workloads were affected, with delays of up to an hour. No data was lost, and the systems recovered once the issue was resolved.
The failure originated in the job queueing service's job-dispatching component, which encountered unexpected CPU throttling combined with short session timeouts for a Kafka consumer group. The consumer entered a loop, unable to stabilize before timing out and restarting its coordination process. The service kept accepting and recording new work but could not pass jobs to workers. Engineers mitigated the issue by shifting load to the standby service and redeploying the primary service. GitHub has extended monitoring to speed up diagnosis of this failure mode and is pursuing changes to prevent it recurring.
