Push handling at GitHub: from one monolithic job to parallel consumers
A push to GitHub triggers far more than a ref update. Pull request synchronization, webhooks, workflow triggers, app configuration installs, Pages builds, and Codespaces updates are just a sample of the work kicked off by every push. In the GitHub monolith, more than 60 distinct pieces of logic, owned by over 20 services, run in direct response to a single push.
Until recently, all of that logic lived in one giant background job called RepositoryPushJob. When the Rails monolith was notified of a push, it enqueued this job, which executed its tasks in a long sequential chain. That design created several serious problems, all stemming from the job’s size and entanglement.
The cost of a monolithic push job
Retrying the enormous job was nearly impossible in practice. Different tasks had conflicting retry requirements: writing Push records to the database can be safely retried at any time and will gracefully handle duplicates, whereas push webhooks are time-sensitive and should never be sent twice. Since a job retry restarts everything from the beginning, these incompatible concerns made retries dangerous. As a result, most steps were never retried at all, and much of the logic was wrapped in catch-all error handlers so a single failure wouldn’t kill the whole job. This meant crucial processing steps could silently never run.
The coupling also created a large blast radius. Even though most tasks rescued errors, a few early steps did not. For instance, writing data to the Pushes MySQL cluster happened near the start of RepositoryPushJob. Any later step—including pull request synchronization, which has no need for that cluster—implicitly depended on it. Failures in that database cluster could take down unrelated processing and cause incidents.
Finally, sequential execution hurt latency. Tasks at the end of the chain waited for all prior logic to complete, adding potentially a second or more of delay for user-facing work like pull request synchronization.
Decoupling into parallel, owned jobs
The fix was to break the long sequential process into many isolated, parallel ones. GitHub added a new Kafka topic, with an event published for every push. Each push-processing task was examined and grouped by owning service and logical relationships—order dependencies, retry-ability, and similar concerns. Each coherent group became its own background job, with a clear owner and appropriate retry configuration. These jobs are enqueued by independent consumers that respond to each Kafka event, using an internal system for enqueueing background jobs in response to Kafka messages.
Supporting this architecture required several platform investments:
- A reliable Kafka publisher that retries until broker acknowledgement.
- A dedicated pool of job workers for the new fan-out queues.
- Enhanced observability to monitor the flow of push events and identify bottlenecks.
- A system for consistent per-event feature flagging, allowing gradual rollout and rollback between the old and new pipelines without data loss or double processing.
Measurable improvements
The new architecture delivers benefits across the entire push lifecycle, directly addressing each problem of the old design.
Smaller blast radius. An issue in one piece of push handling can no longer take down unrelated processing. Around 300 million push processing operations run per day in the new pipeline that previously depended on the Pushes MySQL cluster; they now have no such dependency. Decoupling also redistributed code ownership: what was one owning team is now 15 or more appropriate service owners. New push functionality can be iterated on without unintentional cross-team impact.
Lower latency. Parallel execution means no task waits on unrelated steps. Pull request synchronization in particular shows a notable decrease in completion time:
Improved observability. Smaller jobs offer a cleaner view of each task’s behavior, enabling monitoring that is much more finely scoped and making it easier to pinpoint push-processing problems.
Better reliability. Each job now has retry configuration tailored to its own small set of concerns, without re-executing unrelated logic. Under the old system, a push was “fully processed”—meaning all desired operations completed with no failures—about 99.897% of the time. In the worst-case estimate, the new pipeline fully processes 99.999% of pushes.
