The 45-minute CI bottleneck

In measuring Lead Time for Changes—how long code takes from commit to running in production—GitHub’s engineering team found a serious drag: every commit required about 45 minutes of CI before a pull request could be merged. With a second full run required before deploying a merge branch, a developer could easily wait close to two hours before a change reached GitHub.com. That wait time has now been cut to 15 minutes.

The Ruby monolith behind GitHub.com currently has more than 7,000 test suites and over 5,000 test files. Every pull request commit triggers 25 CI jobs, of which 15 must pass before merge. Each commit consumed roughly 45 minutes and 600 cores of compute. The jobs fell into four groups: unit testing, linting/performance, integration testing, and builds/deployments. All but two integration testing jobs finished within 13 minutes. Those two were the bottleneck—and some of the related suites were flaky, compounding the friction.

A performance-focused tradeoff

Those two long-running jobs existed to verify that changes to GitHub.com would not regress the GitHub Enterprise Server experience. GitHub Enterprise Server ships a patch release every two weeks and a major release quarterly, so this coverage was important—but the suites gave no additional value to continuous GitHub.com deployments. The challenge was balancing Enterprise Server quality against developer velocity on GitHub.com. The result was the deferred compliance tool, integrated with the CI workflow system.

How deferred compliance works

Under deferred compliance, the two long-running integration jobs are no longer required to pass before a pull request is merged. Instead, the tool watches those jobs:

  • If a CI job fails, GitHub creates an issue labeled deferred compliance, tagging the pull request author and code owners for the affected segment.
  • A Slack warning goes to the developer, and a 72-hour timer starts.
  • The developer has 72 hours to fix the build, push a change, or revert the pull request.
  • A successful run automatically closes the compliance issue and stops the timer.
  • If the job stays broken past 72 hours, deployments to GitHub.com are halted—barring exceptional situations—until Enterprise Server integration tests are fixed.

The 72-hour window is customizable, but that value was chosen deliberately: it ensures a Friday-afternoon merge by someone in San Francisco does not block a Monday-morning deployment for a developer in Sydney. The tool is not limited to these two suites—it can be used for any long-running CI job that doesn’t need to gate deployments but still requires accountability for failures.

Outcomes

  • Internal tooling can enforce reliability while reducing manual overhead, giving developers both autonomy and guardrails.
  • Tracking one key metric—lead time for changes—made the bottleneck obvious and pointed toward a simple fix.
  • Understanding why legacy decisions were made is as important as the decision itself; it leads to better, less blunt-force engineering design.

The result is a modest but meaningful win: CI runs that previously took 45 minutes now complete in 15, and the new tooling gives developers real ownership over keeping Enterprise Server healthy without turning every commit into a wait.