Git 2.45 opens the door to the reftable reference backend

Git 2.45 is out, bringing with it a long-awaited milestone: preliminary support for the reftable reference storage backend. The release includes changes from more than 96 contributors, 38 of whom are new to the project. While the full changelog is substantial, the reftable integration stands out as the headline feature.

References in Git — the branches and tags that point to commits — have historically been stored in two ways. They are either "loose" files under $GIT_DIR/refs, such as $GIT_DIR/refs/heads/my-feature, or entries in the single $GIT_DIR/packed_refs file. For most repositories this arrangement works fine, but at scale it begins to show strain. A large number of loose references means directories with many entries, slowing lookups and risking inode exhaustion. Meanwhile, maintaining a single packed_refs file becomes costly: even a small reference update requires rewriting the entire file, an I/O-heavy operation that grows with the total number of references.

Reftable offers a different approach. It is a binary format designed to address these scaling problems, and was originally created by Shawn Pearce for JGit to support Gerrit's very large reference counts. The format's goals include:

  • Near constant-time lookup for individual references, as well as near constant-time verification that a given object ID is referenced at least once.
  • Efficient namespace-wide reference lookups via prefix compression.
  • Atomic reference updates whose cost scales with the size of the update itself, not the total reference count.

At a high level, a reftable-based repository stores references across any number of *.ref files, each organized into variable-sized blocks. Those blocks may hold reference data directly or point to contents in other blocks when references span multiple blocks. The design prioritizes both minimal storage footprint — reference names are prefix-compressed — and fast lookups, even when reading .ref files from a cold cache.

Because the format allows multiple *.ref files to coexist, each reference update transaction can be handled individually without touching existing files. A separate compaction process describes how adjacent *.ref files are merged into one to keep read performance high over time.

Git 2.35 first brought an implementation of the reftable format into Git's codebase, but at the time Git could not use it with its existing reference backend system. That means you could not actually create a repository that stored references with reftable. Git 2.45 changes that: reftable-powered storage is now integrated into Git's generic reference backend system, making it possible to experiment with the format directly.

To initialize a new repository with the reftable backend, use the --ref-format=reftable flag:

$ git init --ref-format=reftable /path/to/repo
Initialized empty Git repository in /path/to/repo/.git
$ cd /path/to/repo
$ git commit --allow-empty -m 'hello reftable!'
[main (root-commit) 2eb0810] hello reftable!
$ ls -1 .git/reftable/
0x000000000001-0x000000000002-565c6bf0.ref
tables.list
$ cat .git/reftable/tables.list
0x000000000001-0x000000000002-565c6bf0.ref

Alternatively, for an existing repository you can rely on the same integration to convert and play with reftable on your own terms:

$ git init --ref-format=reftable /path/to/repo

For readers interested in the full technical details, the original reftable specification is documented in the Git source tree. The Git 2.45 release includes a substantial series of commits integrating this backend into the reference backend framework, laying groundwork for what will likely become a much more prominent feature in future releases.

Experimental SHA-1/SHA-256 Interoperability Takes Shape

Git’s long-running transition from SHA-1 to SHA-256 has been a recurring theme in our coverage of the project. For newcomers: Git identifies every object—blobs, trees, commits, tags—by a hash of its contents. SHA-1 has served that role since the beginning, but known collision attacks like Shattered and Shambles mean an attacker can craft two distinct inputs sharing the same SHA-1 hash. (Some hosts, including GitHub, deploy SHA-1 implementations that reject inputs carrying the fingerprints of such colliding pairs.)

The Git project began sketching a migration in 2017 and settled on SHA-256 as the replacement. Experimental SHA-256 support landed in Git 2.29 (October 2020) and was declared stable in Git 2.42 (August 2023). One of the transition’s stated goals has been letting SHA-1 and SHA-256 repositories talk to each other, so a repository could use one hash locally while pushing to a remote using the other.

Git 2.45 delivers an early, experimental step toward that goal with a new “compatibility” object format. You can now refer to objects by their native hash or by their compatibility hash—the hash the object would have gotten under the other function. A quick walkthrough:

$ git init --object-format=sha256 /path/to/repo
Initialized empty Git repository in /path/to/repo/.git
$ cd /path/to/repo
$ git config extensions.compatObjectFormat sha1

Create a simple commit containing a README file with “Hello, world!”:

$ echo 'Hello, world!' >README
$ git add README
$ git commit -m "initial commit"
[main (root-commit) 74dcba4] initial commit
 Author: A U Thor <[email protected]>
 1 file changed, 1 insertion(+)
 create mode 100644 README

As expected, cat-file shows the commit’s root tree ID computed with SHA-256:

$ git rev-parse HEAD | git cat-file --batch
74dcba4f8f941a65a44fdd92f0bd6a093ad78960710ac32dbd4c032df66fe5c6 commit 202
tree ace45d916e870ce0fadbb8fc579218d01361da4159d1e2b5949f176b1f743280
author A U Thor <[email protected]> 1713990043 -0400
committer C O Mitter <[email protected]> 1713990043 -0400

initial commit

But with git rev-parse told to use the compatibility hash, you can ask for the same commit’s SHA-1 object ID. Its root tree OID is now a different value (7dd4941980 instead of ace45d916e) computed under SHA-1:

$ git rev-parse --output-object-format=sha1 HEAD
2a4f4a2182686157a2dc887c46693c988c912533

$ git rev-parse --output-object-format=sha1 HEAD | git cat-file --batch
2a4f4a2182686157a2dc887c46693c988c912533 commit 178
tree 7dd49419807b37a3afd2f040891a64d69abb8df1
author A U Thor <[email protected]> 1713990043 -0400
committer C O Mitter <[email protected]> 1713990043 -0400

initial commit

Treat this as a preview: much of the machinery for full interoperability remains unfinished, and behavior may surprise you. Even so, this release marks a meaningful first step on that roadmap.

source

Debugging Missing Objects, Even at the Tips

Anyone who scripts around a repository has used git rev-list, which lists commits reachable from a set of inputs. It is also a diagnostic tool for corruption and missing objects—git rev-list --missing=print lists objects that are reachable but absent locally.

The problem: if a missing object sits at the tip of your query itself—for instance, a corrupt branch or tag—Git stops immediately:

$ git rev-parse HEAD | tr 'a-f1-9' '1-9a-f' >.git/refs/heads/missing
$ git rev-list --missing=print --all | grep '^?'
fatal: bad object refs/heads/missing

The command refuses to proceed because one input (refs/heads/missing, via --all) is missing. That made it awkward to inspect holes deeper in history.

Git 2.45 lets you perform that same debugging even when the tips are gone:

$ git rev-list --missing=print --all | grep '^?'
?70678e7afeacdcba1242793c3d3d28916a2fd152

source

A Few More Fixes Worth Knowing

git reflog list

Reference logs, or reflogs, answer questions like “what was main pointing at two weeks ago?” Each reference carries its own reflog, viewable via git reflog (for the checked-out branch) or git reflog refs/heads/some/branch.

Listing which references have reflogs was easy with the files backend—just look at .git/logs:

$ find .git/logs/refs/heads -type f | cut -d '/' -f 3-

With reftable, however, reflogs are stored in a binary format, so a simple find won’t cut it. Git 2.45 adds git reflog list to list references with reflogs regardless of backend.

source

Custom Diff Prefixes

Git’s diff output uses the prefixes a/ and b/ to mark the “before” and “after” sides of each file:

$ git diff HEAD^ -- GIT-VERSION-GEN
diff --git a/GIT-VERSION-GEN b/GIT-VERSION-GEN
index dabd2b5b89..c92f98b3db 100755
--- a/GIT-VERSION-GEN
+++ b/GIT-VERSION-GEN
@@ -1,7 +1,7 @@
#!/bin/sh

GVF=GIT-VERSION-FILE
-DEF_VER=v2.45.0-rc0
+DEF_VER=v2.45.0-rc1

LF='
'

Two new configuration options, diff.srcPrefix and diff.dstPrefix, let you choose your own labels. Use something like “before” and “after” for clarity, or set the prefix to ./ in a terminal that supports path hyperlinking, making file paths in diffs clickable.

source

Multi-Character Comment Strings

When you write a commit message, Git opens your editor with instructions prefixed by a comment character (default #):

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# On branch main
# Your branch is up to date with 'origin/main.

Since 2013 you could change that character via core.commentChar, which helped when a commit message needed to reference a GitHub issue like #12345—a line starting with #12345 would otherwise be swallowed as a comment.

Git 2.45 generalizes that: core.commentString (a synonym for core.commentChar) now accepts any multi-byte character or even an arbitrary string of characters.

source

git config --comment

Speaking of comments: the .gitconfig format has long allowed trailing comments starting with #. The git config command now has a --comment option to append one when writing a new line:

$ git config --comment 'to show the merge base' merge.conflictStyle diff3
$ tail -n 2 .git/config
[merge]
conflictStyle = diff3 # to show the merge base

That is handy when you tweak an obscure setting and want a reminder of why you chose the value you did.

source

git cherry-pick --empty

During a rebase or cherry-pick spanning many commits, some commits can become “empty”—their changes are already present on your branch. For rebases, --empty accepts drop, keep, or stop to control what happens to such commits.

git cherry-pick had no equivalent, so cherry-picking a long series meant handling each empty commit manually with either git cherry-pick --skip (to drop) or git commit --allow-empty (to keep). Git 2.45 ports --empty to git cherry-pick, so you can state your preference once at the start of the operation.

source

Dig Deeper

This only scratches the surface of what shipped in Git 2.45. For the full rundown, see the 2.45 release notes, or the notes from any prior release in the Git repository.