Two Availability Incidents Hit GitHub in April
GitHub reported two distinct availability incidents during April 2021, one affecting the Container registry within GitHub Packages and a second impacting API requests tied to repository creation.
Container Registry DNS Failures
On April 1, starting at 21:30 UTC, the GitHub Packages Container registry suffered a degraded state lasting one hour and 34 minutes. The root cause was DNS resolution failures that caused intermittent connection errors among internal services supporting the Container registry. With requests unable to resolve, users could neither push new container images nor pull existing ones. Since the Container registry was still in public beta, only beta users saw the impact; the broader GitHub Packages service continued operating normally.
As a mitigation measure, GitHub said it plans to increase DNS resolution cache times so that intermittent resolution failures have less blast radius in the future.
Repository Creation Blocked by Data Migration Bug
A second incident began April 22 at 17:16 UTC and lasted 53 minutes. Service monitors flagged an elevated API error rate that translated into a degraded state for repository creation. Investigation traced the problem to a bug in a data migration designed to isolate secret scanning tables into their own cluster. The migration broke the application's ability to write to the secret scanning database.
The failure surfaced an unforeseen coupling: repository creation depends on secret scanning, which makes a call for every repository created. That dependency meant repository creation was completely blocked until the migration could be rolled back.
GitHub is working with its vendor to update the data migration tool and has revised its migration process to include remediation steps for similar incidents. The application code has also been updated to remove the secret scanning dependency from the repository creation path.
