Incident Summary: November 2024
GitHub experienced a single service degradation during November, affecting notification delivery for roughly an hour.
Notification Service Delays
November 19, 10:56–12:03 UTC
A database host exiting routine maintenance came back in read-only mode, which disrupted the notifications service. As a result, between 10:56:00 UTC and 12:03:00 UTC, no notifications were sent to dotcom customers, with delivery delayed by an average of about one hour.
Engineers resolved the issue by flipping the host back to writable state. This allowed notification delivery to resume, and failed delivery jobs were retried successfully. All notifications were fully delivered and accessible to users by 12:36 UTC.
To reduce the risk of recurrence, GitHub is improving observability across its database clusters. The goal is to shorten time-to-detection for similar failures and to strengthen system behavior during host startups.
