Why large monorepo pushes were failing
GitHub's infrastructure handles tens of millions of developers across hundreds of millions of repositories. But for a handful of customers running very large monorepos—repositories updated by thousands of developers daily—push operations were failing at an unusual rate. GitHub's own github/github monorepo was affected too.
The Git Systems organization launched Project Cyclops, a multi-month effort spanning Git storage, protocols, and client teams, plus upstream Git contributors and web front-end engineers. Early work involved helping customers optimize their usage patterns, such as coalescing multiple pushes into single operations to reduce write transactions. But the core fixes required changes to Git itself, repository maintenance scheduling, and server-side reference updating. The result: push errors dropped to nearly zero for the largest monorepo customers.
Faster repository maintenance
GitHub runs maintenance on a repository after every 50 git push operations or after receiving 40MB of unpacked files. This routine repacks loose objects into packfiles and de-duplicates data. On large monorepos with heavy traffic, 50 pushes accumulate quickly, so maintenance was being scheduled while developers were still actively pushing—and frequently failing to complete within the maximum allowed window.
Two changes nearly eliminated those maintenance failures.
Tuning git repack CPU usage
During maintenance, git repack compresses objects by finding related pairs and storing some as deltas against others. Searching for delta candidates across a sliding window can require CPU-intensive comparisons. GitHub implemented a parameter that limits the number of expensive comparisons performed during a repack. Tuning this value cut CPU time substantially while increasing resulting packfile size only marginally—and eliminated almost all maintenance failures.
Retrying spurious failures sooner
Previously, when maintenance failed for any reason, it wouldn't run again for seven days. GitHub introduced a spurious-failure state for failures caused by heavy push traffic during maintenance. In this state, maintenance is retried every four hours, up to three times, which means retries often land during off-hours when push traffic is low. This change eliminated the remaining maintenance failures and reduced on-call toil.
Removing an artificial push limit
File servers hosting git repositories had a long-standing parameter that throttled the rate of push operations processed per server. It was originally designed to prevent any single customer's writes from monopolizing resources in GitHub's multi-tenant environment. In practice, it acted as an artificial cap on server throughput. After testing with the parameter raised incrementally to allow 100% of pushes to run immediately, GitHub found performance was sufficient and removed the parameter from the code entirely. This immediately improved monorepo performance and eliminated many push-related errors.
Precomputing replica checksums
GitHub writes five replicas of each repository across three data centers. Updating Git references requires briefly taking a lock across all replicas while a checksum is computed on each to verify they're in sync, using a three-phase-commit (3PC) protocol. Incremental checksums normally complete in under 50ms, but during repair operations—when the checksum is recomputed from scratch—the lock on large monorepos could be held for 20-30 seconds.
By computing replica checksums before acquiring the lock, GitHub reduced lock hold time to under one second, allowing more write operations to succeed immediately.
Measured results
Customers running their own internal git performance metrics confirmed the improvement. One large monorepo customer reported zero push failures for months. Another, which had been planning a migration to a fresh repository to reduce reference count and improve push success, cancelled the migration after seeing push failures drop to nearly zero and receiving no recent failure reports from developers.
Remaining work
Some residual push failures stem from random internet networking issues beyond GitHub's control. The team is also refreshing storage hardware and refactoring parts of the Ruby monolith into a new microservice written in Go, which is expected to improve repository performance for all users.
Overall impact
Single-repository update traffic rates improved by at least an order of magnitude. Maintenance failures, wasted CPU cycles on file servers, and push errors for large monorepo customers—including those on GitHub Enterprise Server—have been reduced to near zero, with smaller but noticeable gains for the broader GitHub user base.
