Byte-Pair Encoding (BPE) Tokenization
Represent text as subword tokens learned from corpus frequency — balancing vocabulary coverage against sequence length, handling rare words by decomposing them into known pieces.
Intent & Description
🎯 Intent
Build a vocabulary of subword units that handles rare and unseen words without OOV failures — by decomposing them into known pieces rather than failing or using a generic unknown token.
📋 Context
Word-level tokenization produces massive vocabularies and fails on rare words. Character-level handles everything but produces very long sequences that stress the context window. BPE finds the middle ground — common words become single tokens, rare words decompose into subword pieces.
💡 Solution
Initialize with a character-level vocabulary. Iteratively merge the most frequent adjacent pair of tokens into a new compound token, adding it to the vocabulary. Repeat until the target vocabulary size is reached (typically 32K–100K). Byte-level BPE (GPT-2/GPT-4 tokenizer) starts from raw bytes — guaranteeing zero OOV on any Unicode input.
Real-world Use Case
📌 TL;DR
Subword tokenization handles any text by merging frequent pairs. Smaller vocabulary means longer sequences — evaluate fragmentation on your domain before committing to a tokenizer.
Advantages
- Handles rare and unseen words gracefully through subword decomposition — no OOV
- Vocabulary size is a tunable parameter — balance between sequence length and embedding table size
- Byte-level BPE eliminates OOV entirely — any Unicode input is encodable
Disadvantages
- Domain-specific terms may fragment into many tokens — inflating sequence length and wasting context
- Tokenization is model-specific — mismatching tokenizer to model corrupts input silently
- Vocabulary size trades off sequence efficiency against embedding table memory