Keeping unreachable objects lean with cruft packs

Git classifies every object in a repository as reachable or unreachable. Reachable objects can be found by walking from some reference — a branch or tag — through the commit graph and its associated trees and blobs. Unreachable objects have no such path from a reference.

A repository needs all of its reachable objects to remain intact, but unreachable objects can be safely discarded at any time. Doing so is often desirable, particularly when many of them have accumulated or disk space is tight. Git already does this automatically during garbage collection, but a configuration setting called gc.pruneExpire adds a twist.

That setting defines a grace period during which unreachable objects are left alone, even if they're candidates for removal. The grace period protects against a race condition: an unreachable object that's about to be deleted could become reachable by another process — an incoming reference update or a push — right before it's removed, leaving the repository corrupted.

A small, non-zero grace period makes this race much less likely. But it raises the question of how to track the age of unreachable objects that survive the grace period. Packing them together into a single packfile wasn't an option, because all objects in a pack share the same modification time — updating one object effectively refreshes them all. Prior to Git 2.37, each surviving unreachable object was therefore written out as a loose object, with its mtime recording its age. When many unreachable objects are too new to prune, that approach can create serious problems.

Git 2.37 introduces cruft packs, which let unreachable objects be stored together in a single packfile. Each object's age is recorded in an auxiliary table stored in an *.mtimes file next to the pack. Cruft packs don't eliminate the underlying data race, but they make it much less likely to matter in practice by allowing much longer grace periods without the loose-object explosion.

$ git gc --cruft --prune=1.day.ago

The pack directory will then contain an additional .mtimes file storing the ages of unreachable objects written within the previous 24 hours.

$ ls -1 .git/objects/pack
pack-243103d0f640e0096edb3ef0c842bc5534a9f9a4.idx
pack-243103d0f640e0096edb3ef0c842bc5534a9f9a4.mtimes
pack-243103d0f640e0096edb3ef0c842bc5534a9f9a4.pack
pack-5a827af6f1a793a45c816b05d40dfd4d5f5edf28.idx
pack-5a827af6f1a793a45c816b05d40dfd4d5f5edf28.pack

A filesystem monitor built into Git for Windows and macOS

Working directory size is one of the biggest factors in Git performance. Running git status, for example, can require crawling the entire working directory to determine which files have changed. Git keeps its own cached understanding of the filesystem to avoid full traversals, but refreshing that understanding against the actual disk state can be expensive.

Historically, Git allowed integration with external tools like Watchman through a hook, replacing the expensive refresh with a long-running daemon that tracks filesystem state directly. That approach required installing and configuring a third-party tool, which was cumbersome.

In Git 2.37, this functionality is built into Git itself on Windows and macOS, removing the need for external tooling. Enable it for a repository by setting the core.fsmonitor configuration:

$ git config core.fsmonitor true

The first git status after enabling the monitor takes the normal amount of time. Subsequent commands use the monitored data and run noticeably faster.

Sparse index integrations are complete

Git's sparse index feature speeds up commands when sparse-checkout is in use on a large repository. The sparse index allows the index — which tracks the content of the next commit, which files are modified, and more — to only track the parts of the repository you actually care about.

Pairing the sparse index with partial clones means you can work with just the objects you need while keeping the index itself small. When the sparse index was first introduced, different Git subcommands had to be updated individually to take advantage of it. With Git 2.37.0, all of those integrations have landed in the core Git project and are available to everyone.

The final integrations in this release cover git show, git sparse-checkout, and git stash. The git stash work delivers the largest performance improvement of any integration so far — since the command reads and writes indexes multiple times in a single process, it achieves a nearly 80% speed-up in certain cases.

Sparse checkout changes

Git 2.37 deprecates the non---cone style of sparse checkout pattern definitions. In non-cone mode, users specify individual files using a .gitignore-style syntax. That flexibility comes with real costs: matching all patterns against all files can cause slowdowns, and the mode is incompatible with the sparse-index, which provides the main performance benefits of sparse checkouts. The non-cone style still works, but the Git project now encourages --cone mode for all new sparse checkout configurations.

Batched fsync

The previous release made fsync behavior more configurable. Git 2.37 adds a new strategy to the core.fsyncMethod option: batch. This mode improves performance when Git writes many loose objects. Instead of calling fsync() after every write, Git stages updates to the disk’s writeback cache, performs a single fsync() to flush the cache, and then atomically moves files into place. This guarantees durability once objects enter the object directory.

The new mode only applies to loose object writes, and only when core.fsync includes the loose-objects value. On a synthetic test of git add with 500 files, results vary significantly by platform:

  • Linux: 0.06 seconds without fsync(), 1.88 seconds with per-object fsync(), and 0.15 seconds with batched fsync().
  • Windows: 0.35 seconds without fsync(), 11.18 seconds with per-object fsync(), and 0.41 seconds with batched fsync().

Filtered history traversal

Revision-walking commands like log and rev-list support the --since option to limit output to recent commits. That option works by walking commit parents and halting traversal as soon as it encounters a commit older than the specified date. In the presence of clock skew, this can cause Git to omit commits that fall within the time window. Consider three commits where C2 (the parent of C3) has a timestamp from a day ago because of clock skew, while both C1 and C3 are from the last hour. A traversal with --since=1.hour.ago stops at C2 and only shows C3.

Git 2.37 introduces --since-as-filter for those who expect clock skew in their history. This flag filters out commits older than the given date without halting traversal, so commits like C1 still appear in the output.

Remote and configuration improvements

Partial clones use object filters at clone time, but until now there was no straightforward way to see which filter a remote used. The information was buried in the output of git config --list or similar commands:

$ git config remote.origin.partialCloneFilter

In Git 2.37, git remote -v includes this detail directly. The filter appears in square brackets after the remote URL:

$ git remote -v
origin    [email protected]:git/git.git (fetch) [tree:0]
origin    [email protected]:git/git.git (push)

That output makes it easy to see that origin was cloned with a tree:0 filter.

Also new is the transfer.credentialsInUrl configuration, which controls Git’s behavior when it encounters plain-text credentials embedded in a URL within its configuration. Storing credentials this way is discouraged: Git passes full URLs, including credentials, to other programs, and the configuration itself is stored in plain text. Git’s credential mechanism or tools like GCM should be used instead. The new setting takes three values: allow (the default, no action), warn, and die.

Interactive add in C

Git’s interactive mode, git add -i, has been in the process of being ported from Perl to C as part of a long-running effort to move Git commands to C, avoiding sub-process spawn costs on some platforms. The reimplementation has been available in releases since v2.25.0, and in recent versions it ran behind an opt-in add.interactive.useBuiltin configuration. Git 2.37 enables the C version by default, which should result in faster git add -p performance, particularly on Windows.

Developer-focused changes

There is also ongoing work for Git contributors. That includes an improved localization workflow, cleaner GitHub Actions CI output, and reductions in memory leaks within internal APIs. Those interested in contributing can consult the project’s MyFirstContribution guide in the repository’s documentation for getting started.

Release notes

These highlights cover only a portion of what shipped in Git 2.37. The full release notes for 2.37 and all previous versions are available in the Git repository’s Documentation/RelNotes directory.