Why a dedicated code search engine?
GitHub’s code search has come a long way since the platform’s early days, but the underlying requirements have shifted dramatically. The service now faces a corpus of over 200 million repositories, with more than 61 million created in the past year alone. That scale is compounded by velocity: over 170 million pull requests merged annually, plus direct branch pushes, means the index must reflect repository changes within minutes.
Those constraints alone would strain a general-purpose search engine, but code search has special demands that generic full-text search was never designed to meet. Natural language techniques like stemming and tokenization are often counterproductive for source code, where identifier boundaries and punctuation carry meaning. Developers need substring matches, wildcards, and regular expressions. Relevance scoring tuned for web pages or prose does not translate to code, where exact symbol matches and structural proximity matter more than term frequency heuristics. Finally, the latency bar is unforgiving: to become a daily tool, p95 query times must stay well under a second—and most repository- or organization-scoped queries should be far faster.
GitHub’s journey to a bespoke engine started with off-the-shelf systems. Each served a generation of needs, but none could stretch to meet all of these requirements at once.
From Solr to git grep
The first global code search debuted in 2008, when GitHub announced public code search using a Solr index over all public documents. That approach kept access control simple—everything in the index was public—but it left private repositories completely unsearchable.
To fill the gap, repository pages offered a “Search source code” field. Public repositories still hit the Solr index scoped to the current repository, while private repositories fell back to shelling out to git grep. Google Code Search, then in beta, also began crawling public GitHub repositories, giving developers another route for global queries until the service was discontinued a few years later.
The split between public and private search paths proved confusing, and git grep had its own problems. It scans documents without an index, so query time grows with repository size. That could exhaust resources on the Git hosts and force timeouts; large private repositories effectively became unsearchable. A better architecture was needed.
The Elasticsearch era
By 2010, GitHub had started evaluating Elasticsearch as an alternative. Two years later, its potential was clear after a promising experiment indexing gists. In early 2013, as Google Code Search wound down, GitHub launched a redesigned code search backed by an Elasticsearch cluster, consolidating public and private search into one experience. At launch, the index covered almost five million repositories.
The initial rollout was rocky. Within weeks, code search outages hit, and the postmortem revealed a cluster of 26 storage nodes with 2 TB of SSD each, running Elasticsearch 0.19.9 and 0.20.2. The backfill of repository data took several months. Several Elasticsearch bugs were identified and fixed to bring the service back online.
By November 2013, the system had scaled to indexing eight million repositories, handling five search requests per second on average. The Elasticsearch experience has been largely positive since then. Search on GitHub.com in all its forms relies on it, and it continues to perform well.
The code search cluster became the largest GitHub operates. It has grown 20–40x since the 2013 case study, now running 162 nodes with 5,184 vCPUs, 40 TB of RAM, and 1.25 PB of backing storage. It handles 200 requests per second on average and indexes over 53 billion source files. That an off-the-shelf search engine carried this load for so long is a testament to Elasticsearch's capabilities. But the trajectory was clear: only a purpose-built search engine could meet the strict performance and feature requirements ahead.
Why Code Isn’t Text
Code search is fundamentally different from searching through prose. In natural language, punctuation is mostly noise. In source code, those symbols—., :, ;, /, and the rest—carry meaning. Yet GitHub’s Elasticsearch-based code search historically ignored them entirely, as the documentation makes explicit:
You can’t use the following wildcard characters as part of your search query:
. , : ; / \ ` ' " = * ! ? # $ & + ^ | ~ < > ( ) { } [ ] @. The search will simply ignore these symbols.
The root cause was the ingest pipeline’s text analysis. Elasticsearch converts unstructured text into searchable tokens by applying a series of normalizations—case folding, whitespace compression, and tokenization. For code, GitHub configured a custom pattern tokenizer that split on exactly the punctuation characters that queries were forced to ignore:
%q_[.,:;/\\\\`'"=*!@?#$&+^|~<>(){}\[\]\s]_
The resulting tokens went through another round of splitting to break out CamelCase and snake_case subwords. A declaration like pub fn pthread_getname_np(tid: ::pthread_t, ...) became a stream of tokens including pthread, getname, np, and pthread_getname_np. The punctuation itself was never indexed.
This approach was a deliberate trade-off between index size, query performance, and the set of queries you can answer. It worked well enough to launch and evolve code search for nearly a decade, but it created sharp edges. A search for thread_getname against rust-lang would return nothing, even though pthread_getname_np exists in a repo under that org. The tokenization split on the underscore and never produced a token for just thread, so the substring was simply absent from the index. Power users quickly hit these limits and reached for regex tools like git grep.
Chasing Better Tokenization
The limitations were known early. Internal discussions dating to October 2012—more than a year before public launch—weighed alternatives, including trigram tokenization as described by Russ Cox. The verdict at the time was that trigrams would give excellent search results but demanded too much in search time and index size for the then-current Elasticsearch cluster. The team launched with the best-effort identifier-based tokenization instead.
An idea that resurfaced around 2016, after conversations with Elasticsearch experts at Elasticon, was to use a Lucene tokenizer pattern with lookahead/lookbehind assertions. Splitting on transitions between word and non-word characters creates a token for each symbol, allowing queries like answer >= 42 to match source text containing that expression. Experiments showed the cost clearly: indexing time increased by 43–100%, index size grew 18–28%, and typical query slowdown sat at 2.1x, with some queries up to 4x slower.
By 2019, scaling investments in the Elasticsearch cluster opened up headroom, and GitHub Universe 2019 saw the announcement of an exact-match search beta. It followed the tokenizer idea above and projected a 1.3x increase in Elasticsearch resource usage. The beta was allow-listed to specific repositories and orgs, and while illuminating, it proved hard to reconcile with ongoing corpus growth. Worse, the approach still had no path toward substring or regex searches. The beta was sunset in August 2020, after just over half a year.
Project Blackbird
The decision to sunset exact-match search was shaped by a promising research prototype that had been running internally since early 2020. Code-named Blackbird, it was built to determine what technologies could actually deliver the full wish list: comprehensive indexing of all code on GitHub, incremental updates, document deletion, fast exact-match and regex queries with a p95 under a second globally, and all within resource budgets comparable to the existing Elasticsearch cluster.
No off-the-shelf solution fit. Russ Cox’s trigram index for codesearch stores only document IDs in posting lists, which keeps it compact but degrades quickly as the corpus grows. Successor projects adding positional information or other data impose storage and RAM costs that don't scale to GitHub's corpus—Zoekt reports a typical index size of 3.5x corpus size. Sharding strategy also matters: repo- or org-based sharding creates uneven load, and per-repo overhead becomes prohibitive at GitHub's scale.
Blackbird convinced the team to build a custom engine in Rust. The key design decision is sharding by Git blob object ID, which yields deduplication savings and uniform load distribution. The index also stores symbol definitions and supports regex over document content. It is compact—roughly two-thirds the size of the deduplicated corpus—and fast for real-world searches, though pathological queries can still miss the index.
With a single index covering all of GitHub, ranking is critical. You must surface useful documents first. Blackbird’s heuristics are a mix of code-specific signals (ranking definitions higher, penalizing test code) and general-purpose ones (preferring complete matches over partial ones, so a search for thread ranks thread above thread_id above pthread_getname_np). Repository popularity also factors in, so results from widely-used open-source projects appear before matches in obscure test repos.
This is still work in progress. Scoring, ranking, index optimization, and the query language are all under active iteration, with a long feature backlog ahead. But the decision was to get the current capabilities into users’ hands and let feedback guide what comes next.
Built on Open Source
None of this would be close to where it stands without the open source ecosystem. Specific acknowledgments go to:
- The Rust, Go, and React communities and frameworks that underpin the implementation.
- @BurntSushi, whose
regexandaho-corasickcrates were invaluable. - @lemire’s work on fast bit packing and optimization techniques, particularly around SIMD.
- Enry for language detection and Tree-sitter for symbol extraction, both of which power core parts of Blackbird.
