Bringing ShopifyQL to CodeMirror

ShopifyQL Notebooks, launched in October 2022, gives merchants a guided code editing experience for analyzing shop data with ShopifyQL. The editors rely on CodeMirror, a web-based code editor framework. Since CodeMirror ships without ShopifyQL support, the engineering challenge was connecting the two—without rewriting the existing language infrastructure.

ShopifyQL's grammar is defined in ANTLR and shared across client and server targets (currently TypeScript and Go). Protobuf types are also shared between targets. On the front end, all language features are encapsulated in a TypeScript-based language server built on ANTLR's TypeScript target. That server conforms to Microsoft's Language Server Protocol (LSP), which standardizes features like tokenization, parsing, completion, hover tooltips, and linting. Editors and language servers that both speak LSP become interoperable by definition.

The Adapter Problem

CodeMirror uses its own parser engine, Lezer, to generate the parse trees that power editor features. Lezer ships with grammars for common languages but supports no ShopifyQL grammar and does not conform to LSP. Since the ShopifyQL grammar and language server already existed in ANTLR, rewriting them as a Lezer grammar made little sense. Instead, the team built an adapter that speaks LSP on one side and integrates with Lezer on the other. The adapter passes a ShopifyQL query to the language server, adapts the response, and returns a Lezer parse tree.

Lezer supports tree creation in two ways:

  1. Manually attaching nodes in the correct tree shape.
  2. Generating a tree from a buffer of tokens.

The ShopifyQL language server can emit a stream of tokens from a document, making the second approach the natural fit: reshape that stream into a buffer Lezer understands.

From Query to Parse Tree

Transforming a ShopifyQL query into a Lezer tree follows a fixed sequence:

  1. Lezer initiates parse tree creation—on initial document load and on every document change.
  2. The custom adapter passes the ShopifyQL query to the language server.
  3. The language server returns a stream of tokens describing the query.
  4. The adapter converts those tokens into Lezer node types.
  5. The node types form a buffer describing the document.
  6. The buffer builds a Lezer tree.
  7. The tree returns to Lezer, completing the parse cycle.

The Offset Problem

The trickiest part was the token format. The ShopifyQL language server returns tokens as integers in chunks of five, where each position has distinct meaning: length, token type, and token modifier were straightforward. But the line and start-character values behaved unexpectedly. Tokenization is incremental, so each computed offset is relative to the previous token, not to the document start.

A simple query like SHOW product_title shows the behavior: the token for product_title sits on line 1 (zero-based), yet its line integer is zero because it's relative to the previous token. Adding five spaces before SHOW changes only SHOW's start character (from 0 to 5); product_title's values stay untouched since the whitespace between the two tokens didn't change.

Comments make this even more confusing. In some ANTLR grammars, comments go on a separate channel parsed only after the main channel is done. If a comment appears two lines above the last parsed token, the parser must move the pointer backward to tokenize it—hence negative values for the line integer (e.g., -2).

CodeMirror treats offsets more simply: everything is relative to the top of the document, viewed as one long string. Newlines and whitespace are significant and shift token start offsets.

The TokenIterator Solution

Bridging the two offset systems required converting ANTLR-style relative offsets into CodeMirror-style absolute offsets. The solution was a custom TokenIterator that follows the "directions" of the language server's offsets and converts them along the way.

At a high level, the TokenIterator:

  • Derives each line's length from the document, ensuring trailing whitespace is properly represented.
  • Internally tracks the current line and character position.
  • Ingests ANTLR-style line, character, and token-length descriptors and moves the internal position accordingly.
  • Computes the CodeMirror-style start offset from the current position and line lengths.
  • Combines start offset with token length to compute the end offset.

Enabling Editor Features

With the offset conversion solved, the parse tree construction path was complete: feed the query to the language server, tokenize, convert to a Lezer buffer, and build the tree. CodeMirror then understands ShopifyQL well enough to provide syntax highlighting.

But the language server offers more: code completion, linting, and hover tooltips. CodeMirror's plugin ecosystem provides the integration points. The team adapted the language server's doValidate to CodeMirror's linting plugin, doComplete to the autocomplete plugin, and doHover to the requestHoverTooltips plugin.

With these connections in place, the ShopifyQL editor delivers an assistive experience: syntax highlighting, autocomplete, inline diagnostics, and contextual hover information for merchants.

The approach kept a single ANTLR grammar serving both client and server while giving CodeMirror the parse trees it needs. Because the solution integrates with CodeMirror's internal tree structure, subsequent editor decisions can build on a solid foundation rather than work around it.