Git 2.39: Flexible history summaries and safer garbage collection

The open source Git project has released version 2.39, bringing contributions from over 86 developers (31 of them first-time contributors). Among the notable changes: a more expressive git shortlog, improvements to object pruning that help recover from corruption, faster searches in sparse repositories, and a connectivity check that scales better on servers with many hidden references.

Aggregating commits with custom groups

Most command-line Git users are familiar with git log, but its companion git shortlog is less well known. It summarizes what git log produces; many projects, including Git itself, use git shortlog -ns to generate the contributor lists that accompany release announcements.

$ git shortlog -ns v2.38.0.. | head -10
   166  Junio C Hamano
   118  Taylor Blau
   115  Ævar Arnfjörð Bjarmason
    43  Jeff King
    26  Phillip Wood
    21  René Scharfe
    15  Derrick Stolee
    11  Johannes Schindelin
     9  Eric Sunshine
     9  Jeff Hostetler
  [...]

Git 2.29 introduced the --group option to git shortlog, which made it possible to group commits by fields beyond the author or committer. For example, a command could count commits by author while also attributing them to anyone listed in a Co-authored-by trailer.

$ git shortlog -ns --group=author --group=trailer:co-authored-by

In this release, git shortlog goes a step further: --group now accepts arbitrary formatting specifiers from the pretty formats documentation. That opens up aggregation along any dimension Git can format. Want to see how many commits landed each month of a release cycle? Previously you would have had to pipe git log output through a chain of tools, formatting the date as YYYY-MM and counting unique values:

$ git log v2.38.0.. --date='format:%Y-%m' --format='%cd' | sort | uniq -c

Now Git does the work for you. The same summary is produced with a single command:

$ git shortlog v2.38.0.. --date='format:%Y-%m' --group='%cd' -s
     2  2022-08
    47  2022-09
   405  2022-10
   194  2022-11
     5  2022-12

The -s flag instructs git shortlog to emit a summary, with the commit count on the left and the group identifier (here, a year-month combination) on the right. Since any pretty format specifier is valid, the grouping possibilities are bounded only by the format language itself.

External backups for pruned objects

Readers following Git's garbage collection work will remember the introduction of cruft packs in Git 2.37. Running git gc with the cruft option splits your objects into two packs: reachable objects, and unreachable ones touched within a recent grace period. That window is what keeps git gc from racing with incoming reference updates, which could otherwise corrupt the repository as objects are deleted before the refs pointing to them arrive.

$ git gc --cruft --prune=5.minutes.ago

That mechanism is effective in practice but not foolproof. When corruption does occur, having a nearby copy of the discarded objects can make recovery straightforward. Git 2.39 gives git repack a new --expire-to option that works alongside --cruft. Unreachable objects older than the grace period are written to a packfile in a directory you specify, before they are pruned:

$ git repack --cruft --cruft-expiration=5.minutes.ago -d --expire-to=../backup.git

With that command, unreachable objects untouched for more than five minutes are gathered and stored in a pack under ../backup.git. A subsequent garbage collection won't destroy the only copy of objects you might later need. This behavior mirrors the "limbo repository" approach that powers GitHub's own garbage collection; those patches were under review at the time that technique was documented, and now they ship in mainline Git.

One related change: enabling feature.experimental at the bleeding edge of Git now makes cruft packs the default for git gc, instead of requiring an explicit --cruft flag each time.

Faster grep with sparse indexes

Commands working inside a sparse checkout have been gradually taught to operate without expanding the index. One holdout was git grep --cached, which searches the index rather than working-tree blobs; in large repositories it first expanded the index, adding noticeable latency before any results appeared.

A Google Summer of Code participant, Shaoxuan Yuan, removed that requirement. When your search stays inside the sparse cone, the improvement in Git 2.39 can be dramatic — roughly a 70% speedup over the previous release for a restricted git grep --cached search.

Scaling push validation on large servers

The connectivity check has been the subject of optimization in this release, with a payoff for server operators in particular.

Before accepting a push, a Git server advertises its known branches and tags so the client can omit objects the server already has. Once the new objects arrive, the server performs a "connectivity check": verifying that no new object references something nonexistent, which would corrupt the repository. Entering those new objects should never cause inconsistency, so the check is fundamental to maintaining data integrity.

Git servers can be configured to hide certain references from advertisement, but previously all references — hidden and visible — were still checked for connectivity. When a server has a very large number of hidden refs, that extra work adds significant processing latency.

In Git 2.39, the connectivity check only considers advertised references plus those being pushed. In a test repository containing almost 7 million references (only about 3% of them advertised), this change makes Git 2.39 process incoming pushes roughly 4.5 times faster than its predecessor.

Security hardening

Git 2.39 rounds out with defense-in-depth changes. git apply now refuses patches larger than approximately 1 GiB, preventing potential integer overflows in the apply code path. Additionally, trace output from GIT_TRACE_CURL=1 or GIT_CURL_VERBOSE=1 now correctly redacts sensitive header information when HTTP/2 is in use.

This release also saw substantial work on documenting how Git coordinates and discloses embargoed security releases. If you're interested in how that process works — or how to participate — the details live in Documentation/howto/coordinate-embargoed-releases.txt in the Git repository, alongside the SECURITY.md reporting guide.

Complete details on all changes are available in the release notes for 2.39.