June Overview

GitHub experienced four incidents in June that significantly impacted or degraded the availability of several GitHub.com services. Additionally, this report provides further findings from an incident that occurred in May affecting multiple services.

Actions Startup Delays: June 1

At 09:37 UTC, GitHub Actions noticed a significant increase in the time required to start customer jobs. On-call engineers were paged and the service was marked red. Investigation revealed that the pods running the database proxy server were crash-looping due to out-of-memory errors. A change to increase memory for these pods was fully rolled out by 10:08 UTC, with recovery beginning even earlier. Actions moved to yellow at 10:17 UTC and returned to green by 10:28 UTC. The incident spanned 48 minutes in total.

The underlying cause was traced to data analysis queries pointed at an incorrect database. The resulting load triggered the crash loops and broader impact. Those queries have since been moved to a dedicated analytics setup that doesn't serve production traffic. GitHub is adding alerts to detect load increases on the proxy server and investigating guardrails to ensure production database access is limited to services that own the data.

Copilot Authentication Errors: June 21

Shortly after the general availability of Copilot, users with Marketplace or Sponsorship plans couldn't use the service. These users received errors from the API responsible for creating authentication tokens, impacting slightly less than 20% of active users at the time.

Elevated error rates were first noticed around 16:45 UTC. After identifying the issue and confirming impact with a red status, engineers rolled out a fix to the API endpoint. Error rates began declining around 17:45 UTC. By 18:00 UTC the issue was resolved, but the team waited 10 additional minutes before returning to green to check for regressions. Additional testing has been added around this combination of subscription types, and data shape testing will be expanded before future rollouts.

Pending Investigations

Two incidents in late June remain under investigation, with full details expected in the July Availability Report.

  • June 28, 17:16 UTC (26 minutes): Alerting systems detected degraded availability for Codespaces.
  • June 29, 14:48 UTC (1 hour 27 minutes): GitHub Actions, API Requests, Codespaces, Git Operations, Packages, and Pages were all impacted. Contributing factors and remediation efforts will be covered in the July report.

Follow-Up on May 27 Incidents

Both disruptions on May 27 — at 04:26 UTC (21 minutes) and 07:36 UTC (1 hour 21 minutes) — stemmed from the same underlying factors. In the first instance, a service team observed higher-than-normal load and elevated error rates on API requests, particularly at the login endpoint. The impact did not constitute widespread outage; in hindsight, a yellow status would have been more appropriate than red. After the load pattern subsided, the situation was reported as mitigated.

Three hours later, sustained high traffic again degraded service, again concentrated on the login endpoint. All services were statused red due to sustained error rates across a range of clients and situations, then individual services were updated to yellow based on SLO compliance while unaffected services returned to green. The second wave of impact lasted roughly 15 minutes. Elevated traffic continued until a network-level mitigation was applied, after which affected services returned to green.

Data from the incident informed additional application-side mitigations for sustained load of this type and architectural changes to improve resilience. On-call procedures are also being revised to ensure status levels always reflect SLO review, with red reserved for cases where it's truly warranted.