When a Codebase Outgrows the IDE

Developer tools—go-to-definition, code search, documentation generation, linting, dead-code detection—all depend on information extracted from source code. That job is called code indexing. An IDE can index on demand when you open a project, but at Meta's scale that approach breaks down. C++ compile times alone make startup indexing impractical, and when thousands of engineers are working in the same monorepo, repeating the index work on every machine is wasteful. The data also becomes too large to ship around, so it needs to live on a server and be queried over the network.

The result is a centralized architecture with a few moving parts: indexing jobs that run in parallel, a distributed query service that spreads client load, and replicated databases backed up centrally.

General-Purpose by Design

Glean, open-sourced by Meta in August 2021, isn't the first code indexing system. Formats like LSIF have long cached code-navigation data for IDEs. But Glean was built to avoid two constraints: being tied to a specific language, and being tied to a specific use case. The design goals were to store whatever data a language needs and to let users query it in a general way.

That generality shows up in two core choices:

  • Schemas are language-specific but user-defined. Glean doesn't dictate what data you can store. Most indexed languages have their own schema, and the system can hold arbitrary non-code data too. Underneath, data is persisted in RocksDB for scalability and efficient retrieval.
  • The query language is logic-based and declarative. Called Angle (an anagram of Glean), it can derive information automatically—either on the fly at query time or ahead of time. That lets Glean offer a language-neutral view layered on top of language-specific data, similar to how SQL views abstract over tables.

Storing detailed, language-specific data has practical payoffs. In C++, for example, Glean's data includes which using statement resolves each symbol reference. That detail is what makes accurate unused-include and unused-using detection possible. Meanwhile, a client that just wants a code outline can query the higher-level, language-agnostic layer without caring about C++ specifics. There's no need to choose one or the other.

Angle: Schema and Query in One Language

Angle is Glean's single language for both schema definitions and queries. A schema fragment for C++ function declarations is a set of type definitions—records with fields and types. A FunctionDeclaration is a predicate (think SQL table), and its instances are facts (think rows). Queries return facts, and you can query efficiently by specifying a prefix of the fields, such as the name of a function.

To fetch a specific function like folly::parseJson, a query specifies the name and namespace; results come back in about a millisecond. Angle handles more complex patterns too, like finding all classes that inherit from a class named exception and have a method what that overrides a base-class method. Because there may be many results, the query server supports incremental fetching, returning the first hits in a few milliseconds.

Incremental Indexing: O(changes), Not O(repository)

A monorepo's index is perpetually in danger of being stale. As the repository grows in size and rate of change, full re-indexing takes so long that the data lags the latest commits by hours. The obvious solution—only process what changed—is deceptively hard to implement.

Two problems stand in the way. First, storing destructive updates makes it impossible to serve multiple revisions simultaneously without duplicating entire databases. Second, true O(changes) behavior is unattainable for most languages. Modify a C++ header, and every file that includes it, directly or transitively, must be reprocessed. That expansion is called the fanout, so the practical target is O(fanout).

Glean handles the first problem by stacking immutable databases. Layers sit on top of each other; each can add or hide facts relative to the layer beneath, yet the stack presents itself to clients as a single database. Revisions coexist without extra full-size copies.

The second problem—calculating fanout—is language-specific. For C++, Glean can derive it through queries: find all files that include one of the changed files, and repeat that query until nothing new turns up. That keeps incremental indexing as close to O(changes) as the language allows.

Code navigation and beyond

Glean's indexing model offers advantages over IDE-based code navigation, particularly for large monorepos. The key difference is that indexing happens ahead of time, so navigation is instantly available in any web-based code browser without waiting for IDE or language server initialization. This also enables full-repository visibility: finding all references to a function, not just those visible in an open project. That capability supports dead code detection and helps locate clients of an API slated for change.

The architecture for code navigation centers on Glass, a symbol server that wraps Glean's complexity behind a straightforward API. The code browser makes a single documentSymbols(repo, path, revision) call to receive all definitions and references in a source file, including source and target spans. Definitions render as a file outline, references as navigable underlines. Other features like Find References and Call Hierarchy are driven by additional Glass calls. Glass itself is open source and available in glean/glass on GitHub.

Glean also fills a gap for IDE users working on very large projects. For C++ developers at Meta, the language service blends Glean-provided data with native clangd results: since Glean has already analyzed the whole repository, go-to-definition, find-references, and doc comment hovercards work immediately on startup. The approach targets C++ first because long compile times make it the worst IDE experience, but it is not language-specific.

Because Glean stores full API details — classes, methods, signatures, inheritance — and collects documentation from standard language conventions (like /// or /** */ comments in C++), it can generate on-demand API documentation. A sample page for the folly::Singleton type shows the result:

These pages are produced by Glass and rendered client-side, with fully hyperlinked navigation across all APIs in the repository.

Symbol IDs and change analysis

Every symbol gets a unique symbol ID from Glass — for example, REPOSITORY/cpp/folly/Singleton. The ID provides a stable URL for documentation even if the definition moves, and can be used to query for references across the repo. The exact format varies by language.

Glean also indexes diffs to generate a diff sketch: a machine-readable summary of changes, such as newly introduced classes, removed methods, or new function calls. These sketches drive simple static analysis for potential issues, support non-trivial lint rules and rich notifications, and enable semantic search over commits — for example, linking a production stack trace to recent commits that modified the affected function. Indexing diffs also powers code navigation in code review tools like Phabricator, giving reviewers go-to-definition and type-on-hover on the changes they are examining. This is enabled for multiple languages including C++, Python, PHP, Javascript, Rust, Erlang, Thrift, and Haskell.

Other uses and extensibility

Glean sees use in a range of other applications: analyzing build dependency graphs, detecting dead code, tracking API migrations, measuring code complexity metrics, managing test coverage and selection, and automating data removal. It also supports Retrieval-Augmented Generation (RAG) in AI coding assistants.

The system is deliberately designed for queries and data that have not yet been anticipated. Ad-hoc queries from engineers and automated systems solve a broadening set of problems, so Glean aims to be as general as possible both in what data it stores and what questions it can answer. More details and getting-started documentation are available on the Glean site.