Reliability Engineering for a Monolith on a Friday

GitHub runs a monolithic application, and deploying it is meant to be an invisible event. The company ships to production dozens of times a day—Fridays included—using two primary targets: Kubernetes clusters for the customer-facing application and bare metal hosts for other workloads. Because both targets differ in runtime and replica counts, the tooling that coordinates the process is a critical piece of infrastructure in its own right.

A slow or fragile deployment pipeline costs more than outage risk. Hundreds of engineers work on the same codebase, trying to merge and ship features. If the pipeline is slow or requires manual intervention, developers spend more time babysitting deployments than writing code. To keep the process healthy, the engineering team had to answer three questions with data, not intuition:

  • How long does code take to go from merge to running in production?
  • How often do deployments get rolled back?
  • How frequently does a deployment need manual help?

Instrumenting the Pipeline

The first step was measuring the entire workflow. The team added metrics around CI build time, the duration of each pipeline step, total deployment time, final pipeline state, rollback counts, and retry events. They also tracked the aggregate delivery throughput: the number of weekly merges and pull request deploys, as well as the time between a PR becoming deploy-ready and actually merging.

Those metrics did more than surface bottlenecks. They revealed classes of intermittent failures that could be tolerated rather than treated as fatal. The team introduced automatic retries for those cases, which improved reliability even before the root causes were addressed. The instrumentation also serves as an early warning system, letting the team spot problems sooner than they would through ad hoc debugging.

Reducing Toil with Better Tooling Output

Kubernetes deploys for GitHub involve starting hundreds of pods across multiple clusters. The previous deployment tooling rarely surfaced what Kubernetes was doing behind the scenes. A failure not caught by earlier canary stages would force the engineer to interrogate the cluster directly—asking someone to understand the internals of Kubernetes, which GitHub engineers are not expected to know.

The infrastructure team looked at internal support requests and found a pattern of preventable toil. By analyzing those requests, they shipped tooling changes that give engineers more detail while a deployment runs. If a rollout fails, tooling now proactively surfaces granular Kubernetes-level information, including relevant events, without the engineer needing cluster access. This new level of visibility shortens the time to root-cause a failure, which feeds directly into the overall deployment time.

Treating Deploys as an SLO

Failures in GitHub deployments do not typically translate to customer impact. Automated halts coupled with Kubernetes readiness probes stop a bad rollout before it reaches users. But the real problem becomes throughput: the longer a deployment takes, the fewer deployments that day has room for.

To guard against that, the team defined SLOs around deployment health. These are intentionally focused on how fast code reaches production, rather than on consumer-facing success rates or latency. The objectives now act as a steering signal for the team responsible for continuous deployment. When the numbers trend in the wrong direction, the team knows to prioritize improvements to the shipping flow over new features in the tooling.

Since adopting this measurement-driven approach, the team has made meaningful progress on deployment speed and reliability. More importantly, the SLOs convert ephemeral complaints about the deployment process into a defined baseline—one that helps the team decide when to invest in the pipeline so engineers can ship the rest of the year without friction.