A Pure-Go SentencePiece BPE Tokenizer

Earlier this year, Eli Bendersky published a post on implementing BPE tokenization in Go, replicating OpenAI's tokenizer. Building on that, he has now released go-sentencepiece: a pure-Go implementation of the SentencePiece tokenizer used by Google's Gemma and Gemini models. The canonical implementation is C++ with Python bindings via SWIG; cgo wrapping is possible, but a pure-Go approach removes the C compiler dependency, which is the motivation for this new library.

Two important limitations should be noted. First, while SentencePiece supports both BPE and Unigram tokenizers, go-sentencepiece implements only the BPE variant, as that is what the Google models actually use. Second, it covers only the encoding and decoding phases, not the training phase; on that topic, the author refers readers to his previous post on BPE implementation.

Screenshot of tokenizer with a sample text, showing tokens

SentencePiece's design differs from OpenAI's BPE variant in two fundamental ways:

  1. Whitespace handling. Text is not pre-split by whitespace using regex; instead, whitespace is regular input with its own token IDs. Spaces are typically represented by the "fat underscore" character (U+2581). Single spaces are usually attached to the following non-space token, but multi-space sequences exist as distinct tokens. This difference has a significant impact on algorithm performance.
  2. Configuration. SentencePiece tokenizers rely on a full protobuf configuration, not just a vocabulary and regex. go-sentencepiece currently supports the subset of options used by Google AI's models, but adding more is straightforward.

Performance-Driven Algorithmic Changes

The original BPE encoder from the earlier post used simple quadratic-time algorithms, which were acceptable because they operated on single words. SentencePiece processes entire texts, making that approach impractical. To handle full-text inputs efficiently, go-sentencepiece adopts techniques from the C++ implementation:

  • A trie data structure in the prefixmatcher package, used to match string prefixes against the set of possible tokens.
  • A generic heap-based priority queue in the priorityqueue package, used to determine which token pair to merge next.

These algorithmic choices alone yielded roughly a 100x speedup in encoding, enough that performance should not be a bottleneck in practice, even without fine-grained micro-optimization.

Setup and Configuration

Because SentencePiece is protobuf-configured, the library requires two parts. The first is the .proto schema file, vendored into the internal/model directory from the C++ repository. A pre-generated .pb.go file is included, so the protobuf compiler isn't needed unless the schema changes. The second part is the actual protobuf data file—the tokenizer model with its vocabulary—which can be downloaded from the official Gemma PyTorch repository. go-sentencepiece loads this model file directly.

Test It Online

As with the earlier project, the tokenizer has been compiled to WebAssembly with HTML/JS scaffolding, offering a live demo. The page is at https://eliben.github.io/go-sentencepiece/; the model file is large, so expect a few seconds of load time on slower connections.