When One Repository Isn’t Enough
Treating Git as the distributed database behind your engineering system invites a familiar question: what do you do when the database gets too big? For application databases, the common answer is sharding—splitting the data across multiple nodes. For Git, the equivalent problem appears in the form of sprawling repositories, and the solutions follow a similar logic of partitioning the data, though the mechanics differ.
Application databases often shard horizontally using a shard key, typically a sortable string so related records land in the same shard. Git’s object database defies that approach: object IDs are hashes of their contents, so their prefixes are effectively random. Instead, sharding a repository means splitting on structural lines: logical components, directories in the worktree, or chunks of history.
Breaking Out Components: The Multi-Repo Pattern
A straightforward way to shard a database is to isolate independent tables in their own services. The Git analog is splitting a large repository into several unrelated repositories, each representing a logical slice of the system—commonly, a microservice extracted from a monolith. Each resulting repository lives on its own, with no explicit links to the others.
This works best when each repository has a dedicated team with end-to-end ownership of development, deployment, and monitoring. The separation mirrors database sharding where each shard is tucked behind a single component’s interface; other parts of the system don’t need to know it exists.
The trade-off is human overhead. Discovering how repositories connect requires documentation and tribal knowledge, which makes cross-cutting efforts like security audits harder to track. Shared dependencies become a recurring pain: they must flow through package managers rather than version control, obscuring the set of consumers and weakening test coverage when core libraries change.
A super-repository approach tackles some of those drawbacks by gathering smaller repositories under one roof.
Stitching Repositories: Submodules and Super Projects
Git submodules let one repository—the super repository—embed links to other repositories at designated paths in its worktree. The .gitmodules file records the submodule’s location, and the tree entry at the submodule path points to a specific commit in that submodule’s history.
Each submodule retains its own commit history, refs, and object store, plus its own remotes for synchronization. Cloning the super repository doesn’t pull submodules automatically; you opt in per submodule. This design makes the super repository a de facto registry for small repos in a multi-repo setup, akin to a shard coordinator routing queries to the right shards.
That central hub enables global guarantees. For instance, continuous integration can enforce that a submodule advances only when all cross-submodule builds and tests pass in the super project. That protection guards against core components breaking downstream consumers.
It also creates friction. Submodule repositories are less autonomous than isolated ones. Updates pushed to a submodule’s remote aren’t “complete” until the super project bumps its path pointer to the new commit. This raises a policy question: should a submodule move forward before the super repository validates the change? The answer requires deliberate coordination, particularly when a source dependency spans multiple submodules and a breaking change in one component forces updates in others before all can be merged into the super project.
The complexity has spawned tooling built atop git submodule—Google’s repo coordination tool is a notable example of managing change across multiple submodules.
The One-Repository Answer: Monorepos
Both sharding strategies above reduce repository size by limiting the files in each worktree, but they circulate around a stubborn problem: code dependencies between components that must be resolved at build time. A monorepo sidesteps that entirely by containing all the source needed to build and ship a large system in one place. A practical rule: if it ships together, it merges together.
A popular variant is the service-oriented architecture (SOA) monorepo, where all services share a repository but deploy independently. Here, a component can be tested against the latest versions of its sibling services prior to release, avoiding the multi-repo synchronization nightmare.
The main cost is repository growth. Rapid expansion pushes monorepo users toward Git’s advanced performance features—sparse-checkout and partial clone among them—to keep client clones and checkouts manageable. The build system bears its own burden: without incremental builds, every change would retest the entire system. Successful monorepo shops typically dedicate a team to developer experience and build orchestration.
Yet even the best optimization has limits. Repositories can grow faster than the tooling adapts, at which point it becomes worthwhile to revisit a split—resetting the repository to a smaller, more tractable size. The right strategy depends on the trade-offs you’re willing to make between component autonomy, coordination complexity, and raw repository scale.
Sharding by time
When a monorepo grows too fast for a simple split by worktree, it can instead be treated as a time-series database: the valuable dimension is not what changes but when. A time-based shard cuts the repository at a point on the trunk, keeping the full worktree but discarding everything committed before that point.
The procedure starts with a freeze: pick a moment when trunk can be locked and all merging paused. This is disruptive and should be rehearsed. From that frozen tip, create a new repository whose root commit has the same tree as the old trunk but no parents. The new root commit's message should point back to the old repository and its exact tip commit. All in-flight topic branches then need to be replayed. A straightforward path is to rebase each branch onto the frozen trunk commit, export the commits with git format-patch, and import them into the new shard with git am.
Once the new shard is live, the old repository becomes read-only. New work continues only in the new location, which means CI/CD secrets and repository URLs must be updated across the organization. If the build and deploy pipeline is managed as infrastructure as code, most of that configuration carries over automatically.
The key advantage of time-based sharding is that it works regardless of worktree layout. Unlike splitting into multiple repositories or adopting submodules, no restructuring of the codebase is required. It is also useful when history carries anti-patterns such as large binary blobs, even if the current tree has been cleaned up. The new shard effectively starts fresh, with a much smaller object store than a full history rewrite would produce.
But that old history still matters. Without it, the new repository appears to have been created in one giant commit, and answering questions like “why is this code here?” becomes impossible. Git can bridge the two repositories so history queries cross the boundary, though at a cost. You need a local clone of each shard. Add the old repository's object directory as an alternate of the new one by appending its path to .git/objects/info/alternates. Then use git replace to substitute the new root commit with the old tip commit. Since the two commits share the identical tree, Git can walk from the new history “through” the replacement into the old commits.
Running with replace objects enabled is slower, and it disables some optimizations such as the commit-graph file described earlier in this series. Combined mode should therefore be used only when history queries across the shard boundary are essential. One tactic is to store the replace reference under a non-standard base via GIT_REPLACE_REF_BASE, so normal Git operations never see it and the cost disappears. Setting the variable just for queries that need the full history enables on-demand access.
This combined view also smooths the migration for developers still working on branches from the old repository. With replace references active, git rebase can move those branches onto the new history, removing the need for format-patch and am transformations. A worked example of this style of shard, starting from the v2.37.0 tag of the Git repository, is available in a public sharded repository with instructions for cloning both halves and exercising the combined history.
Moving data to cheaper storage
Another angle is to keep a single repository location but offload the least-used objects to slower storage, much as a database might move cold rows to a cheaper tier. Partial clone already does some of this: a blobless clone created with git clone --filter=blob:none downloads all commits and trees but fetches blob contents on demand. The initial clone is smaller and faster, but every checkout or git blame that needs a missing blob requires a network round trip.
The scheme can be improved by pairing a blobless clone with an alternate. If a full copy of the remote lives on a local network share or on a drive provisioned by IT, pointing the alternate at that copy gives Git a nearby fallback for blob lookups. Unlike a network fetch, the access is on a local network or even on the same machine. This setup also preserves the performance of indexes such as commit-graph and changed-path Bloom filters, making it an attractive way to seed new machines.
Still, none of this changes what happens after the clone. Future fetches keep growing the working repository and never shrink it. The variant that actually offloads is more aggressive: maintain an alternate that references a secondary storage area, then move infrequently used objects off the fast store into that alternate and delete them locally.
Recency is the natural criterion for deciding what to keep. Commit objects are cheap and support most history queries, so they should stay local along with each root tree and any trees reachable from recent roots. Older blobs can move out first, with trees following at increasing depths as commits age. The exact policy is flexible; no existing tool implements this kind of recency-based evacuation from a Git object store, but the space is open for a purpose-built solution tuned to a particular monorepo's access patterns.
