In a git repository, where do your files live?

Where your files actually live inside .git

Understanding git means knowing where its data physically resides. We know everything is in the .git directory, but where do the actual snapshots of your files go? For instance, a file called content/post/2019-06-28-brag-doc.markdown in this repository — and its older revisions — can be located with a little forensic digging.

$ find .git/objects/ -type f | wc -l
2761

Objects are content-addressed

A short Python script reveals the storage scheme. It reads a file, prepends a small header, and computes a SHA-1 hash of the combined bytes.

import hashlib
import sys

def object_path(content):
    header = f"blob {len(content)}\0"
    data = header.encode() + content
    digest = hashlib.sha1(data).hexdigest()
    return f".git/objects/{digest[:2]}/{digest[2:]}"

with open(sys.argv[1], "rb") as f:
    print(object_path(f.read()))

That hash is the object’s filename. In this case the result maps to .git/objects/e3/3121a9af82dd99d6d706d037204251d41d54. Running the script against our sample file produces:

$ python3 find-git-object.py content/post/2019-06-28-brag-doc.markdown
.git/objects/8a/e33121a9af82dd99d6d706d037204251d41d54

This is content addressed storage: the name of an object is a direct function of its contents. A practical consequence: if fifty files have identical contents, git stores that data only once. All fifty files point to the same object path.

Reading the raw object data

Inspecting the object’s raw bytes is confusing until you realize the data is compressed:

$ cat .git/objects/8a/e33121a9af82dd99d6d706d037204251d41d54
x^A<8D><9B>}s<E3>Ƒ<C6><EF>o|<8A>^Q<9D><EC>ju<92><E8><DD>\<9C><9C>*<89>j<FD>^...

The system utility file confirms it:

$ file .git/objects/8a/e33121a9af82dd99d6d706d037204251d41d54
.git/objects/8a/e33121a9af82dd99d6d706d037204251d41d54: zlib compressed data

A tiny zlib-based decompressor reveals the underlying format:

import zlib
import sys

with open(sys.argv[1], "rb") as f:
    content = f.read()
    print(zlib.decompress(content).decode())

$ python3 decompress.py .git/objects/8a/e33121a9af82dd99d6d706d037204251d41d54 
blob 16673---
title: "Get your work recognized: write a brag document"
date: 2019-06-28T18:46:02Z
url: /blog/brag-documents/
categories: []
---
... the entire blog post ...

That output shows blob 16673\0 followed by the complete file contents. The number is the byte length. There are no diffs here — even though this is the ninth revision of that post, the object stores the entire file as of that commit, not a delta from the prior one.

Finding an older version

Locating a historical copy requires traversing commit metadata. First, find the commits where that file changed:

$ git log --oneline  content/post/2019-06-28-brag-doc.markdown
c6d4db2d
423cd76a
7e91d7d0
f105905a
b6d23643
998a46dd
67a26b04
d9999f17
026c0f52
72442b67

Trusting that the referenced commit is stored directly under .git/objects fails in this case — the directory for it is empty. The repository keeps most objects packed to save space, a layer we previously didn’t need to touch. For the time being, it’s worth unpacking everything so we can inspect individual objects.

$ mv .git/objects/pack/pack-adeb3c14576443e593a3161e7e1b202faba73f54.pack .
$ git unpack-objects < pack-adeb3c14576443e593a3161e7e1b202faba73f54.pack

That process balloons the object count from about 2,700 to roughly 20,000:

find .git/objects/ -type f | wc -l
20138

The commit tree walk

Commits, like blobs, are objects. The commit object itself is a small text file holding metadata.

$ python3 decompress.py .git/objects/02/6c0f5208c5ea10608afc9252c4a56c1ac1d7e4
commit 211tree 01832a9109ab738dac78ee4e95024c74b9b71c27
parent 72442b67590ae1fcbfe05883a351d822454e3826
author Julia Evans <[email protected]> 1561998673 -0400
committer Julia Evans <[email protected]> 1561998673 -0400

brag doc

The same data is cleaner with git cat-file -p 026c0f52. That output references a tree — essentially a directory listing with hashes instead of inodes — whose ID is 01832a9109ab738dac78ee4e95024c74b9b71c27.

$ python3 decompress.py .git/objects/01/832a9109ab738dac78ee4e95024c74b9b71c27
b'tree 396\x00100644 .gitignore\x00\xc3\xf7`$8\x9b\x8dO\x19/\x18\xb7}|\xc7\xce\x8e:h\xad100644 README.md\x00~\xba\xec\xb3\x11\xa0^\x1c\xa9\xa4?\x1e\xb9\x0f\x1cfG\x96\x0b

The tree’s contents are cryptic because the embedded hashes are raw bytes, not hex strings. Reading tree objects is easier with git’s own formatter, which shows the archived root directory listing:

$ git cat-file -p 01832a9109ab738dac78ee4e95024c74b9b71c27
100644 blob c3f76024389b8d4f192f18b77d7cc7ce8e3a68ad	.gitignore
100644 blob 7ebaecb311a05e1ca9a43f1eb90f1c6647960bc1	README.md
100644 blob 0f21dc9bf1a73afc89634bac586271384e24b2c9	Rakefile
100644 blob 00b9d54abd71119737d33ee5d29d81ebdcea5a37	config.yaml
040000 tree 61ad34108a327a163cdd66fa1a86342dcef4518e	content <-- this is where we're going next
040000 tree 6d8543e9eeba67748ded7b5f88b781016200db6f	layouts
100644 blob 22a321a88157293c81e4ddcfef4844c6c698c26f	mystery.rb
040000 tree 8157dc84a37fca4cb13e1257f37a7dd35cfe391e	scripts
040000 tree 84fe9c4cb9cef83e78e90a7fbf33a9a799d7be60	static
040000 tree 34fd3aa2625ba784bced4a95db6154806ae1d9ee	themes

Note the lingering accidentally committed mystery.rb. The target file lives under content, so the walk continues through that tree (id 61ad34108a327a163cdd66fa1a86342dcef4518e):

$ git cat-file -p 61ad34108a327a163cdd66fa1a86342dcef4518e

040000 tree 1168078878f9d500ea4e7462a9cd29cbdf4f9a56	about
100644 blob e06d03f28d58982a5b8282a61c4d3cd5ca793005	newsletter.markdown
040000 tree 1f94b8103ca9b6714614614ed79254feb1d9676c	post <-- where we're going next!
100644 blob 2d7d22581e64ef9077455d834d18c209a8f05302	profiler-project.markdown
040000 tree 06bd3cee1ed46cf403d9d5a201232af5697527bb	projects
040000 tree 65e9357973f0cc60bedaa511489a9c2eeab73c29	talks
040000 tree 8a9d561d536b955209def58f5255fc7fe9523efd	zines

While content has only one subtree, resolving the post/ path requires one more cascade:

$ git cat-file -p 1f94b8103ca9b6714614614ed79254feb1d9676c	
.... MANY MANY lines omitted ...
100644 blob 170da7b0e607c4fd6fb4e921d76307397ab89c1e	2019-02-17-organizing-this-blog-into-categories.markdown
100644 blob 7d4f27e9804e3dc80ab3a3912b4f1c890c4d2432	2019-03-15-new-zine--bite-size-networking-.markdown
100644 blob 0d1b9fbc7896e47da6166e9386347f9ff58856aa	2019-03-26-what-are-monoidal-categories.markdown
100644 blob d6949755c3dadbc6fcbdd20cc0d919809d754e56	2019-06-23-a-few-debugging-resources.markdown
100644 blob 3105bdd067f7db16436d2ea85463755c8a772046	2019-06-28-brag-doc.markdown <-- found it!!!!!

At the bottom of that listing is our target. Its hash is 3105bdd067f7db16436d2ea85463755c8a772046, so the old content is now directly retrievable from .git/objects/31/05bdd067f7db16436d2ea85463755c8a772046. A decompression script confirms the object stores that older draft’s bytes.

$ python3 decompress.py .git/objects/31/05bdd067f7db16436d2ea85463755c8a772046 | head
blob 15924---
title: "Get your work recognized: write a brag document"
date: 2019-06-28T18:46:02Z
url: /blog/brag-documents/
categories: []
---
... rest of the contents of the file here ...

Why git log is slow on a file

This multi-step walk (parse a commit, descend into a tree, repeat for the target path) is exactly what git does per revision when you run git log path/to/file — except it does so for every single commit in history. With thousands of commits and one object lookup per step per commit, the command’s occasional sluggishness starts to make sense.

How many retired blobs accumulate?

Counting tracked files:

$ git ls-files | wc -l
1530

Listing everything in the object store gives the total number of distinct objects:

$ find .git/objects/ -type f | grep -v pack | awk -F/ '{print $3 $4}' | wc -l
20135

Most objects are actually trees and commits, not file content. A small script distinguishes blobs from other object types:

import zlib
import sys

for line in sys.stdin:
    line = line.strip()
    filename = f".git/objects/{line[0:2]}/{line[2:]}"
    with open(filename, "rb") as f:
        contents = zlib.decompress(f.read())
        if contents.startswith(b"blob"):
            print(line)

$ find .git/objects/ -type f | grep -v pack | awk -F/ '{print $3 $4}' | python3 find-blobs.py | wc -l
6713

That computes to 6713 minus 1530, or roughly 5,183 old file revisions kept on standby. Disposing of nothing, git waits downstream in case you want any of them back — another way of saying version control carries a lot of weight underneath the surface. As a postscript, creating the commit for this very article triggered git’s own cleanup, repacking those 20,000 objects back into dense packfiles:

$ find .git/objects/ -type f | wc -l
14