Tokenization: splitting text into tokens with BPE and WordPiece
Tokenization converts text into numeric tokens for language models. Subword methods like byte-pair encoding (BPE) and WordPiece split words into smaller units, keeping the vocabulary compact while handling rare words. BPE iteratively merges frequent character pairs, while WordPiece uses a scoring formula to select merges and finds the longest matching subword during tokenization.
The need for subword tokenization
Neural machine translation typically requires a fixed vocabulary, but translation involves an open set of words. The Sennrich et al. paper proposes encoding rare and unknown words as sequences of subword units, based on the idea that many word types can be expressed through smaller pieces, such as proper names, compound words, and loanwords. This approach improved results on English-German and English-Russian translation tasks.
Subword tokenization splits text into units between characters and whole words. The Hugging Face tokenizer summary explains that this keeps the vocabulary compact while still capturing meaningful parts, so common words stay intact and rare words decompose into subwords. This is especially useful for languages like Turkish, where long words are formed by combining subwords.
Byte-pair encoding
Byte-pair encoding (BPE) was originally a compression algorithm and later adopted for tokenization. The LLM course on BPE describes the training process: after normalization and pre-tokenization, the algorithm computes the unique set of words and builds a base vocabulary of characters. It then iteratively merges the most frequent adjacent pair of tokens.
For example, with the words "hug", "pug", "pun", "bun", and "hugs", the pair ("u", "g") appears most often and is merged first to create "ug". The process continues until the target vocabulary size is reached. During tokenization, the merge rules are applied in order. A character not in the base vocabulary becomes an unknown token, unless a byte-level approach is used.
The tiktoken repository explains that BPE is reversible, lossless, works on arbitrary text, and compresses the token sequence relative to the original bytes. Each token corresponds to roughly four bytes on average.
WordPiece
WordPiece is the tokenization algorithm Google developed for pretraining BERT, as described in the LLM course on WordPiece. It is similar to BPE but selects merges using a score: the pair frequency divided by the product of the individual frequencies. This prioritizes pairs whose parts are less frequent on their own.
Unlike BPE, WordPiece does not store merge rules; it keeps only the final vocabulary. Tokenization finds the longest subword from the beginning of the word that appears in the vocabulary, then continues with the rest. For instance, "hugs" is tokenized as ["hug", "##s"], with "##" marking a continuation.
Variations and implementations
SentencePiece is a language-independent subword tokenizer described in the SentencePiece paper. It can train directly from raw sentences without pre-tokenization, treating text as a raw sequence of Unicode characters and escaping whitespace with a meta-symbol.
The SentencePiece repository provides open-source implementations and highlights features like subword regularization and BPE-dropout to improve model robustness. It also supports both BPE and unigram models.
For models from OpenAI, tiktoken is a fast BPE tokenizer written in Rust and available as a Python package. It aims to be fast and provides an educational submodule for learning about BPE.
| Feature | BPE | WordPiece |
|---|---|---|
| Selection of merges | Most frequent adjacent pair | Score: pair frequency divided by product of parts' frequencies |
| Tokenization method | Apply merge rules in order | Find longest matching subword from the start |
| Merge rules stored | Yes, merge rules are kept | No, only final vocabulary is kept |
| Example models | GPT, Llama, Qwen2 | BERT, DistilBERT |
| Base vocabulary | Characters or bytes | Characters with a prefix like ## for subwords |
Frequently asked questions
Why split text into subwords instead of whole words?
To keep the vocabulary compact while handling rare and unknown words. Common words remain as single tokens, and rare words decompose into subwords that the model can still represent.
How does BPE select which pairs to merge?
It repeatedly merges the most frequent adjacent pair in the current token set, based on counts in the training corpus.
How is WordPiece different from BPE?
WordPiece uses a scoring formula that considers the frequency of the pair relative to its parts, and it stores only the final vocabulary, not merge rules.
What is byte-level BPE?
Byte-level BPE uses 256 byte values as the base vocabulary instead of Unicode characters, so every text can be tokenized without unknown tokens.
What is SentencePiece?
SentencePiece is a language-independent tokenizer that trains directly from raw sentences and treats whitespace as a regular symbol, enabling lossless detokenization.