A smaller index for enormous repositories
Git’s sparse-checkout feature has long been the recommended way to work in a monorepo without checking out every file. By limiting your working directory to the directories you actually need, it’s supposed to make a massive repository feel like a small one. But there’s a gap: the index — Git’s internal staging area — still tracks every file at HEAD, even those outside your sparse-checkout scope. In a monorepo with millions of files, that index can balloon past 180 MB, and every command that reads or writes it pays the price.
The Git Fundamentals team at GitHub has been working on a fix: the sparse index. This feature shrinks the index to only the files within your sparse-checkout cone, so its size scales with your working set rather than the entire repository. Combined with other performance features, the impact is dramatic.
Enabling it is straightforward, provided your repository supports cone-mode sparse-checkout:
git sparse-checkout init --cone --sparse-index
git sparse-checkout set <dir1> <dir2> ... <dirN>
To understand why this matters, consider the performance data from a real monorepo with over two million files at HEAD, where a sparse-checkout definition populates roughly 100,000 files. The team compared three setups: the full index, the sparse index, and a repository containing only the sparse-checkout files (the theoretical best case). The sparse index delivers runtimes close to that ideal — within 60 milliseconds even in the worst case measured — while the full index lags significantly behind.
Inside the index
The Git index is a flat list of every file at HEAD, along with its blob object ID and metadata. You can inspect it with git ls-files:
$ git ls-files
LICENSE
README.md
bin/index.js
examples/fetch/git-fetch-after.png
examples/fetch/git-fetch-after.svg
examples/fetch/git-fetch-after.txt
examples/fetch/git-fetch-before.png
examples/fetch/git-fetch-before.svg
examples/fetch/git-fetch-before.txt
examples/fetch/git-fetch-combined.png
examples/fetch/git-fetch-combined.svg
examples/maintenance/trace.png
examples/maintenance/trace.svg
examples/maintenance/trace.txt
package.json
Take derrickstolee/trace2-flamechart as a concrete example. Many of its files live in an examples/ directory, but the core functionality is in bin/. You can focus your working directory on just the necessary files:
$ ls
LICENSE README.md bin examples package.json
$ git sparse-checkout init --cone --sparse-index
$ git sparse-checkout set bin
$ ls
LICENSE README.md bin package.json
However, even after narrowing the working directory, git ls-files still lists every file in the repository. Using git ls-files --debug reveals why: entries outside the sparse-checkout definition have their filesystem metadata stripped and a SKIP_WORKTREE flag set, telling Git not to materialize them.
$ git ls-files --debug
LICENSE
ctime: 1634910503:287405820
mtime: 1634910503:287405820
dev: 16777220 ino: 119325319
uid: 501 gid: 20
size: 1098 flags: 0
README.md
ctime: 1634910503:288090279
mtime: 1634910503:288090279
dev: 16777220 ino: 119325320
uid: 501 gid: 20
size: 934 flags: 0
bin/index.js
ctime: 1634910767:828434033
mtime: 1634910767:828434033
dev: 16777220 ino: 119325520
uid: 501 gid: 20
size: 7292 flags: 0
examples/fetch/git-fetch-after.png
ctime: 0:0
mtime: 0:0
dev: 0 ino: 0
uid: 0 gid: 0
size: 0 flags: 40004000
(...)
So why keep those entries at all? Git needs to understand what the full tree would look like — for generating commits and for handling the case where the sparse-checkout definition later expands. The index also stores object IDs and permission bits for every path, as shown by the read-cache test helper:
$ test-tool read-cache --table
100644 blob 646521d0d6c070e6f15e0f5828be1127d3b75503 LICENSE
100644 blob b230b3a6e2d81d50dc00177e970a10726b5baf08 README.md
100755 blob 918533d51c7a5f91622311893dcfd40bfd4f43d7 bin/index.js
100644 blob e0f88531b916b92821476760672e8161b9954898 examples/fetch/git-fetch-after.png
100644 blob f4a523cd1acb0a9d2620970ad7a43405d6e305dc examples/fetch/git-fetch-after.svg
100644 blob fc4e30dca5fcb0c3d2031dc82a43d5d644e26b41 examples/fetch/git-fetch-after.txt
100644 blob 15dc889965617df3b5a30cf01e52c491e41c59c1 examples/fetch/git-fetch-before.png
100644 blob 602bd5bcbd815914a035d0d4f0d2a3896f600de2 examples/fetch/git-fetch-before.svg
100644 blob bc40a8e4658d17c35de996f3655e737b85ce7ad9 examples/fetch/git-fetch-before.txt
100644 blob 356cdd36e0d78a62af8b010d25d658054bb6fdc7 examples/fetch/git-fetch-combined.png
100644 blob cc0c23f2c8a822c51a17c46268f38c2268b400ae examples/fetch/git-fetch-combined.svg
100644 blob dfc0893d172d841d971e206461466db935b7c192 examples/maintenance/trace.png
100644 blob a10a876472e46c6ae58e6fc6e2adc64d4dae809b examples/maintenance/trace.svg
100644 blob 8f5e8bfbc44674feb3aa96e0b7bf1bf717495658 examples/maintenance/trace.txt
100644 blob a4599a9e0a01c28a2c0a622457664fc8c55bfdf9 package.json
Conceptually, the index has a nested structure: directories like bin/ and examples/ contain subtrees that link to individual blobs. But in practice, the core index format stores everything as one flat array. The nesting is recovered through an extension called the cache-tree extension, which stores tree object IDs and the ranges of index entries each tree covers. Git commands like git add update this cache-tree so that a subsequent git commit can build a tree object quickly.
Why the index dominates monorepo performance
A flamechart of a single git status command in the two-million-file monorepo shows where time goes:
- Reading and parsing the index from disk (multi-threaded, but still heavy).
- Comparing the working directory against the index, which triggers lazy initialization of hash tables.
- Writing the modified index back to disk — which is single-threaded.
Each of those stages suffers from the sheer volume of index entries. git add and git commit are affected too — they’re supposed to feel instant.
The pain is especially acute for teams that want to consolidate related projects into a monorepo. In the team’s experience, a customer wanted to group dependencies that had previously lived in separate, much smaller repositories — some hundreds of times smaller than the monorepo — into the monorepo itself, relying on sparse-checkout to keep the developer experience unchanged. Instead, users felt the weight of the full index on every command, since their daily work only touched a fraction of the files.
The culprit was always the same: millions of index entries for files those users never opened. Build machines can handle checking the whole tree at pull-request time. Individual developers shouldn’t have to.
The breakthrough came with cone-mode sparse-checkout patterns. Unlike older, file-based patterns, cone mode matches by directory. That design was originally about faster pattern matching, but it also created an opportunity: once Git knows your working set is a cone of directories, the index no longer has to enumerate every file outside that cone.
Collapsing directories into single entries
A sparse index stores more than just file paths with blob object IDs — it can also contain directory paths paired with tree object IDs. Because cone mode sparse-checkout patterns match at the directory level, Git can determine when an entire directory lies outside the checkout cone and collapse all of its contained file paths into one directory entry.
In the derrickstolee/trace2-flamegraph example repository, you can enable the sparse index command and inspect the resulting index contents with test-tool read-cache:
$ git sparse-checkout init --cone --sparse-index
$ test-tool read-cache --table
100644 blob 646521d0d6c070e6f15e0f5828be1127d3b75503 LICENSE
100644 blob b230b3a6e2d81d50dc00177e970a10726b5baf08 README.md
100755 blob 918533d51c7a5f91622311893dcfd40bfd4f43d7 bin/index.js
040000 tree b395192a7adbf21793f9489f3623c117802b2043 examples/
100644 blob a4599a9e0a01c28a2c0a622457664fc8c55bfdf9 package.json
The index now holds a directory entry alongside the four blob entries, as shown below:
These sparse directory entries correspond to directories located just outside the sparse-checkout definition. Each such directory also gets a cache-tree node whose range spans only that single sparse directory entry.
Even the --debug output of git ls-files can show the full details, though this currently requires a --sparse flag implemented in a personal fork of Git; a comparable feature should eventually land in the core Git client.
$ git ls-files --debug --sparse
LICENSE
ctime: 1634910503:287405820
mtime: 1634910503:287405820
dev: 16777220 ino: 119325319
uid: 501 gid: 20
size: 1098 flags: 200000
README.md
ctime: 1634910503:288090279
mtime: 1634910503:288090279
dev: 16777220 ino: 119325320
uid: 501 gid: 20
size: 934 flags: 200000
bin/index.js
ctime: 1634910767:828434033
mtime: 1634910767:828434033
dev: 16777220 ino: 119325520
uid: 501 gid: 20
size: 7292 flags: 200000
examples/
ctime: 0:0
mtime: 0:0
dev: 0 ino: 0
uid: 0 gid: 0
size: 0 flags: 40004000
package.json
ctime: 1634910503:288676330
mtime: 1634910503:288676330
dev: 16777220 ino: 119325321
uid: 501 gid: 20
size: 680 flags: 200000
Notice that this output is no longer truncated, and the sparse directory entry for examples/ is the only one with blank filesystem data. It also carries the same flags value that sparse file entries used before.
Removing index entries and shortening average path lengths combine to shrink the index substantially. In the example monorepo, most users see their index drop from 180 MB to under 10 MB.
Status at a fraction of the time
Returning to the monorepo example, we can run git status again and compare its flamechart against the version with a full index:
With the sparse index, git status completes in under 200 milliseconds, down from 1.3 seconds. The highlighted regions in the flamechart correspond to the portions of the command that walk the working directory — work that is independent of index size. All other processing is slower in the full index case purely because of the index’s size.
The safety net: expanding sparse indexes
Pruning the index at the directory level means a flat path list now contains two kinds of Git objects. Dozens of places in the Git codebase interact directly with the index, and all of those interactions assume every entry points to a blob. To handle that, a compatibility layer was needed: a way to expand a sparse index to an equivalent full one so unintegrated code paths could still run safely.
The ensure_full_index() method handles that conversion. It inspects the list for directory entries and replaces them with contained file entries by traversing the tree objects under each directory, ignoring filesystem metadata. The method is called immediately after parsing the index, so no index interactions happen until sparse directories are removed.
During expansion, entries are scanned in lexicographic order. File entries copy straight to the new list; directory entries are passed to read_tree_at() to iterate over all contained blobs, generating an index entry for each. The final entry list is copied back, and the index is no longer sparse.
With that protection in place, writing the sparse format came next. The convert_to_sparse() method converts a full index in-memory, using the cache-tree extension to determine object IDs for new sparse directory entries. Existing file entries are copied and directory entries inserted as needed.
Guarded, incremental integration
Shrinking the index was just the start. Since the index is compact, reading it from disk beats recreating it from trees—and ensure_full_index() takes longer to expand a sparse index than to read a full one. The real performance win required teaching Git how to operate on sparse directory entries directly.
A new setting, command_requires_full_index, enabled by default, acted as a guardrail. When a sparse index is parsed, this setting triggers ensure_full_index() unless a command has explicitly disabled it. Calls to ensure_full_index() were also inserted before most index interactions, letting developers single out code paths needing integration by setting a breakpoint on the expansion call and tracing the call stack.
The first command integrated was git status—a challenging start, in hindsight, because it uses several index operations common to other commands. That work paid off later: most of the effort for git checkout and git commit integration was already done.
The git diff case
git diff presented two interesting index comparisons. Comparing the working directory to the index is straightforward: while walking the working directory, Git only drills into directories that exist. If sparse directory entries don't appear as real directories, they're never touched. But if a sparse directory entry does exist as an actual directory, ensure_full_index() expands the index. Since that's not desired, sparse-checkout was updated to delete ignored files outside the cone.
Comparing the index to HEAD (i.e., git diff --cached) is trickier. Differences outside the cone can exist—for example, after git reset --soft moves HEAD without changing the index or working directory. This command compares the HEAD root tree against the index. Files in the sparse index compare normally; sparse directory entries fall back to a tree-vs-tree comparison of the subtrees instead, which also lets the walk prune when two subtrees share an object ID.
Merging with the ORT strategy
Three-way merges were a major open question. The default recursive strategy uses the index as a working structure during computation, which was hard to reconcile with sparse indexes. But the ORT strategy, developed by Elijah Newren, sidestepped this: it doesn't use the index at all, relying instead on a recursive tree structure built from the root tree, creating subtrees only for changed paths. It also became the new default in Git 2.34.
Because ORT was index-free, integrating the sparse index into git merge, git rebase, git cherry-pick, and git revert required little more than ensuring the final index was sparse from creation.
Testing across configurations
Testing was built around a new script that starts with a repository containing interesting data shapes and copies it into three configurations: baseline, cone-mode sparse-checkout, and cone-mode with sparse index enabled. Each test case runs identical commands against all three, expecting matching output and working directory results. This caught real behavioral differences, some leading to fixes and others to deliberate changes—for instance, some commands now require a --sparse flag to modify paths outside the cone in Git 2.34.
The Scalar functional tests were also run regularly against development branches. When those caught issues that generic Git tests didn't, similar tests were added to Git itself. An experimental release with early integrations went to select monorepo users, confirming the sparse index improved performance significantly. One key finding: sparse-checkout should delete ignored files outside the cone, since their presence forces a sparse-to-full expansion, negating the benefits. As a bonus, the working directory shrinks further.
Sparse index adoption and integration status
Git’s sparse index support is not universal yet. Commands that haven’t been integrated trigger a compatibility check on the first index read, converting the sparse index into a full one. The integrated commands have been specifically tested and adjusted to work with the sparse format.
The roll-out has been incremental across recent releases. Git 2.32.0 (June) introduced the sparse index format itself. Git 2.33.0 (August) followed with integrations for git status, git commit, and git checkout. Git 2.34.0, slated for November, brings support for git add, git merge, git rebase, git cherry-pick, and git reset. For monorepo users who couldn’t wait, the microsoft/git fork fast-tracked additional integrations—git diff, git blame, git clean, git sparse-checkout, and git stash—into a pre-release.
Judging by typical monorepo workflows, the currently integrated commands cover almost all user needs. Those who adopt the sparse index with these integrations should see a markedly better experience, assuming their repository size and sparse-checkout cone make the optimization worthwhile.
The microsoft/git fork will enable the sparse index by default for monorepo users in its 2.34 release. The core Git project plans to keep it off by default until all integrations land upstream and the implementation has proven stable over several versions.
Roadmap and community feedback
While the essential command integrations are complete, the remaining work is largely upstream: finishing the contributions to core Git and collecting community feedback on the implementations. More command integrations may follow, but the immediate priority is a smooth transition for monorepo users.
The sparse index is currently in a strong state for general use. Anyone running into performance or stability problems can report issues or start discussions in the microsoft/git repository. The sparse index removes a major obstacle to monorepo-scale repositories, and further scaling innovations will build on that foundation.
