Codespaces Availability: August Incident and July Follow-up

GitHub experienced one availability incident in August that significantly impacted Codespaces. The company also used this month's report to share a deeper root-cause analysis for a July incident affecting the same service.

August 29: Multi-region Codespaces Outage

Starting at 12:51 UTC on August 29, GitHub's alerting systems detected a problem affecting most Codespaces customers. Service remained degraded for 5 hours and 40 minutes. As of this report, the investigation into contributing factors is still underway. GitHub plans to publish a detailed update on cause and remediation in the September Availability Report, due out on the first Wednesday of October.

July 27 Incident: Root Cause and Fixes

GitHub has completed its investigation into the July 27 incident that prevented creation and restart of certain Codespaces workloads for nearly 8 hours. During that event, a subset of codespaces in East US and West US running on 2-core and 4-core machine types could not be created or restarted.

Beginning around 21:30 UTC on July 27, GitHub saw a high rate of failures when creating new virtual machines for Codespaces in those two regions. Demand for 2-core and 4-core codespace creations and starts outpaced the rate of successful VM creations, eventually exhausting the underlying VM pools. By 22:29 UTC, those pools were drained and could not keep up, prompting a yellow status. Affected codespaces experienced longer-than-normal start times while waiting for an available VM; many ultimately timed out and failed.

Each codespace runs on an isolated VM for security. The Codespaces platform builds a host VM image on a regular cadence, and all host VMs are instantiated from that base image. The incident began when GitHub's cloud provider started rolling out an update in East US and West US that was incompatible with how GitHub built its host VM image. Diagnosis was complicated by the provider reporting that VMs were successfully created even though critical processes that should have started during VM creation were not running.

Initial mitigations included scaling up VM pools to absorb the elevated failure rate and adjusting timeouts to fail fast on VMs unlikely to succeed. These measures helped but could not fully offset the rising failure rate as the provider's update continued rolling out. The cloud provider recommended that GitHub adjust its image generalization process for compatibility with the update. After implementing that change in the image build pipeline, VM creation success rates recovered, and the backlog of queued codespace requests was eventually fulfilled.

In response, GitHub has taken three follow-up actions:

  • Audited the VM image building process to ensure alignment with the cloud provider's guidance and prevent future incompatibilities.
  • Improved the service logic and monitoring so critical operations during VM creation are verified rather than trusting only the provider's reported VM status.
  • Updated alerting to detect VM creation failures earlier, before user-facing impact occurs.

GitHub states these changes should prevent this class of issue, surface other failure modes sooner, and speed up future diagnosis and mitigation of VM creation errors.

Real-time status updates are available on the status page, with more engineering details on the GitHub Engineering Blog.