Hendrik van Antwerpen, Alexander Neubeck

So many tokens, so little time: Introducing a faster, more flexible byte-pair tokenizer

We released a new open source byte-pair tokenizer that is faster and more flexible than popular alternatives.

HVHendrik van Antwerpen, Alexander Neubeck·December 12, 2024AI & ML