Tokenization
The process of breaking text into smaller units (tokens) that LLMs process. Tokens are typically sub-word fragments, not whole words.
Tokenization converts text into numerical tokens that an LLM can process. Modern LLMs don't operate on raw characters or whole words — they use sub-word tokens that balance vocabulary size with efficient representation.
Common tokenizers include Byte Pair Encoding (BPE) used by GPT and SentencePiece used by Llama. The word "tokenization" might become a single token, while "antidisestablishmentarianism" might split into 4-5 tokens. Common English words usually map to 1 token; rare words and other languages take more.
This matters practically: LLM pricing is per-token, and 1,000 tokens roughly equals 750 English words. Multilingual content (Chinese, Arabic, code) tokenizes less efficiently. Tokenization quirks also cause some LLM failures, like spelling errors and arithmetic mistakes — issues that better tokenization is gradually fixing.