Why language coverage drives everything

Semantic’s analysis pipeline — diff computation, abstract interpretation, and code navigation — starts with parsing source files into a structured representation. That front-end step, collectively called the language support system, has historically been the bottleneck for expanding GitHub’s program analysis to more languages. GitHub now hosts more than 50 million developers, and the goal is to make analysis available to all of them, regardless of language. The cost of manually adding and maintaining parsers kept that coverage limited.

Diagram showing semantic architecture

CodeGen, the new language support system shipped by the Semantic Code team, attacks that cost directly. By automating much of the pipeline and making it more resilient, CodeGen lowers the barrier to adding new languages that Semantic can parse.

What CodeGen automates

The core idea behind CodeGen is to reduce the manual work involved in turning a language specification into a working parser integrated with Semantic’s analysis toolchain. The system handles the repetitive and error-prone parts of the process, from grammar handling to generating the code that feeds into the rest of the analysis stack.

That automation makes the system more robust in two ways. First, it removes many opportunities for human error in parser construction. Second, it makes the maintenance burden lighter when language specifications change, since the pipeline can regenerate the necessary components rather than requiring hand-updated patches.

The result is a language support system where adding a new language is no longer a large, specialized engineering effort. Instead, the process is structured so that the bulk of the work is done by CodeGen, and the remaining steps are simpler and more consistent across languages.

Why Adding Languages Was Painful

Language support in Semantic used to demand two hand-written grammars per language and a multi-step parsing pipeline:

  1. Parsing with tree-sitter: A grammar was hand-written for each language using tree-sitter.
  2. Bridging to Haskell: The haskell-tree-sitter library provided bindings to tree-sitter's C library.
  3. Generalizing syntax: Parse trees were translated into a custom, generalized representation of syntax based on Wouter Swierstra’s Data types à la carte approach. This "assignment" step required writing a second grammar in Haskell to map tree-sitter AST nodes onto these shared à la carte datatypes.
  4. Adding semantics: A polymorphic typeclass, Evaluatable, was implemented for each à la carte syntax to capture what it meant to interpret the syntax.
  5. Handling effects: Modeling control flow also required representing effects like file system state or non-determinism.
  6. Validating: Tests for diffing, tagging, graphing, and evaluation were added.

The Old System's Faults

This process was not only technically involved but suffered from several critical limitations:

  • Brittleness: Each language's assignment code was tightly coupled to its tree-sitter grammar. Any grammar change could break it at runtime without compile-time errors, leading to tedious, manual tracking and creating a system that was accidentally incentivized against making grammar improvements.
  • Missing named children: Tree-sitter syntax nodes lacked named child nodes. Children were ordered lists with no role information, mismatching Semantic's internal representation. This forced more manual assignment work to deal with nodes that can appear anywhere, like comments or Ruby heredocs.
  • Sub-optimal types: The à la carte syntax types were essentially untyped, enforcing only minimal structure on the tree. This meant any subterm could be any element of the syntax, forcing many Evaluatable instances to handle error conditions that could never actually occur. The approach was insensitive to minor linguistic differences between languages.
  • High effort: Maintaining two grammars per language was time-consuming, error-prone, and tedious. The mechanical nature of the work made it difficult to leverage community support for adding languages.

A Generative Redesign

The overhaul centered on shifting from hand-written transformations to auto-generated, strongly-typed datatypes built from schema defined directly in grammars.

  1. Named child fields: The tree-sitter library was updated with a new field function in the grammar API. This lets you retrieve a node's children based on their field name, and all language grammars were updated to use it.
  2. Generated schemas: The parser generation code now produces a node-types.json file, which statically describes the kinds of children you can expect for each node type, including fields. This provides a clear schema for a language's ASTs.
  3. Auto-generated datatypes: Semantic deserializes the node-types.json file and uses Template Haskell to generate language-specific syntax datatypes from it. Nodes are classified as sums, products, named leaves, or anonymous leaves, and are turned into precise product types, like a Python if_statement.
  4. Generic AST building: Instead of hand-writing a second grammar in an Assignment file, Semantic now uses Haskell's generic metaprogramming framework to automatically map the auto-generated datatypes onto tree-sitter's parse trees via its tree cursor API. This returns an AST with metadata like range and span information.

The result is a set of language-specific, strongly-typed datatypes representing the exact sum of syntax possible at any point in the grammar. This provides strong compile-time guarantees about correctness and completeness—a function's name is typed as an identifier, a switch primitive only contains case statements, and so on. This new system materially bypasses the historical engineering effort; it was possible to completely remove our à la carte syntaxes. As proof of the approach, Java and CodeQL were two new languages shipped using precise ASTs generated from the new system.