Kubernetes Pod Health Check Brought GitHub Down for Over Four Hours

GitHub experienced a service availability incident on July 13, 2020, lasting from 08:18 UTC for four hours and 25 minutes. The root cause traces back to a single container exceeding its memory limits within production Kubernetes Pods, which triggered a cascade of failures across GitHub's clusters.

How a Memory Limit Escalated Into an Outage

The chain of events began when Kubernetes marked production Pods as unavailable. Upon investigation, GitHub found that one container inside each Pod was exceeding its defined memory limits and being terminated. While that particular container isn't required for production traffic, Kubernetes requires all containers within a Pod to be healthy for the Pod to be marked as available.

Typically, this failure mode resolves itself within about a minute as the cluster recovers. However, the affected container had an ImagePullPolicy of Always, meaning Kubernetes was instructed to fetch a new container image on every restart. A routine DNS maintenance operation that had been completed earlier had left the clusters unable to reach the image registry, so the Pods kept failing to start.

Attempting to mitigate the situation, GitHub triggered a redeploy. That action made matters worse, as the failure began propagating across all production clusters. Recovery was only possible after restarting the process with cached DNS records, which finally allowed GitHub to fetch container images, redeploy, and restore services.

Planned Improvements

GitHub has identified several areas of focus to prevent a recurrence of this pattern:

  • Enhancing monitoring to ensure Pod restarts won't fail again based on the same failure pattern
  • Minimizing the dependency on the image registry
  • Expanding validation procedures for DNS changes
  • Reevaluating all existing Kubernetes deployment policies

In parallel, GitHub has an ongoing workstream to improve its approach to progressive deployments. The goal is to carefully evaluate the impact of deployments in a more incremental fashion. This effort is part of a broader engineering initiative focused on reliability, with more details expected in the coming months.