Why GitHub Reworked Its On-Call Model

As GitHub's product surface expands, the company has had to rethink how it runs on-call for its main Ruby on Rails monolith. The previous setup, where one rotation covered the entire codebase, created a set of operational problems that the engineering organization needed to solve.

The core issues with the old monolithic on-call structure were straightforward:

  • Engineers rarely had deep knowledge across the massive monolith, so most pages turned into escalations to another team. The person on-call often felt like a dispatcher rather than an engineer.
  • The rotation was so large that individual engineers were only on-call about four days per year, in 24-hour shifts. That frequency was too low to build real familiarity with the system or its failure modes.
  • Monitoring and runbooks for a monolith-wide rotation were owned by nobody in particular. Alerts were noisy, and documentation was stale.
  • A small group of roughly 5–10 platform experts ended up involved in nearly every production incident, creating an unfair concentration of toil.

Splitting Ownership Along Service Lines

The central change was to break the single monolith rotation into separate rotations so that each engineering team owns on-call for the code it maintains. Getting there required solving both technical and organizational problems.

Mapping Files to Teams

The monolith contains more than 16,000 files, and ownership was often undocumented. GitHub introduced a system that maps files to services, then assigns services to teams. The mapping lives directly in the monolith:

## Apps
### API
app/api/github_apps.rb     :apps
app/api/grants.rb          :apps
### Components             :apps
app/component/apps*        :apps
test/app/component/apps*   :apps

## Authzd
**/*authzd*                :authzd
app/models/permissions.rb  :authzd

Files point to services like apps or authzd. A separate file in the monolith then associates every service with an owning team:

apps:
  name: GitHub Apps
  description: Allows users to build 'GitHub Apps' on top of GitHub.
  team: ecosystem-apps

That data feeds into GitHub's internal Service Catalog, which makes ownership searchable by engineers, support, and product staff. The catalog entry for the apps service shows the team, its associated files, and related operational context:

To enforce adoption, GitHub added a linter that blocks updates or additions to monolith files unless ownership metadata is complete. The company also worked with engineering leadership to assign owners for the major services that were still unclaimed.

Reassigning Alerts and Coordinating Dozens of Teams

Monitoring had been designed for the monolith as a whole. Teams were asked to build alerting scoped to their own services. After most of that work was done, senior engineers reviewed the remaining monolith-wide alerts, routed them to the appropriate teams, and retired the ones that no longer served a purpose.

The coordination effort spanned more than 50 engineering teams. GitHub opened a GitHub issue for each team with a checklist of required changes, then followed up with regular check-ins to keep teams moving and offer help where they were stuck.

The Harder Part: Culture and Training

The logistical work was significant, but the cultural and educational obstacles were tougher. Several factors made the transition particularly demanding:

  • The pandemic hit mid-project. Roughly seven months into the rollout, COVID-19 arrived, adding anxiety that affected engineers' ability to think critically. The project required a more empathy-first, high-touch approach than originally planned.
  • Many engineers had never been on-call. GitHub ran three rounds of training, held office hours with experienced on-call engineers, built tooling and documentation for teams, and opened Slack channels for questions and help.
  • Work-life balance was a real concern. Engineers new to on-call often struggled to imagine routines like grocery shopping or exercise while carrying a pager. GitHub worked with teams to document strategies from experienced on-call engineers, offered 1:1 support, and reinforced that teammates can cover a shift briefly or act as backup. The company's global remote workforce also helps, allowing follow-the-sun rotations where teams can hand off across time zones.
  • Blamelessness is still a work in progress. Engineers worried about missing pages or making mistakes. Leadership reinforced that outages happen regardless of performance, but GitHub acknowledges this is a long-term effort. The company is working on creating safe spaces to learn about on-call and publicly celebrating engineers who took on unfamiliar work during an incident.
  • Criticality varies by team. Some services require a 5-minute response; others can wait until the next business day. Some engineers viewed this as unfair, but GitHub sees it as a natural fit for different interests and tolerances for operational pressure, with engineers self-selecting into teams that match their preferences.
  • On-call improvement needs dedicated time. Teams were concerned they couldn't invest in making their own on-call better. GitHub's guidance: the person on-call, when not responding to pages, should be improving the experience by updating runbooks, tuning alerts, automating repetitive tasks, and fixing technical debt. The company suggested teams might spend roughly 20% of their time on technical debt and 20% on on-call experience improvements when needed. Sustaining that requires consistent messaging from senior VPs down to line managers.

Measuring Progress and Keeping the Effort Going

After the bulk of the rollout, GitHub saw improved incident resolution times. The work isn't finished, though. The company is conducting a retrospective in January to assess how it can roll out large organizational changes better in the future, and it is surveying engineers regularly about their on-call experience while continuing to monitor uptime.

The cultural pieces tied to blamelessness, training, and work-life balance will need reinforcement for years. GitHub plans to keep meeting with teams to identify on-call pain points and share operational best practices, with the goal of making sustainable on-call a fixture of the engineering organization rather than a one-off initiative.