How LLM Tokenization Works (and Why AI Can't Spell)

A large language model never sees the word "strawberry" as ten letters. Before any neural network runs, a separate program called a tokenizer chops the text into chunks called tokens and replaces each chunk with an integer. The model only ever sees those integers, which is why it can write a sonnet and still miscount letters. This explainer on LLM tokenization shows how byte-pair encoding builds the chunk vocabulary, what changes between tokenizers, and how the design choice leaks into spelling, arithmetic, price, and even model bugs.

Why models do not read letters

A neural network can only do arithmetic on numbers, so text has to be converted first. The simplest option would be to give every character its own number and feed the model one character at a time. That works, but the sequence gets very long, and a transformer (the architecture behind modern chatbots) pays a steep price for length. Its attention mechanism compares every position in the input with every other position, so the number of comparisons grows with the square of the sequence length.

The opposite option is to give every word its own number. Sequences get short, but the vocabulary explodes: every name, plural, typo, and technical term needs an entry, and any word the system has never seen becomes a hole. Sennrich, Haddow, and Birch described this problem for machine translation in their 2016 ACL paper, which framed translation as an open-vocabulary problem and proposed encoding rare words as sequences of smaller subword units instead.

Tokenization sits between those extremes. A token is a piece of text, anywhere from a single character to a whole common word, that appears in a fixed vocabulary the tokenizer learned in advance. Common words tend to get one token. Rare words get split into several familiar pieces. Nothing is ever entirely unknown.

An analogy: a Lego set, not a bucket of sand

Think of the three options as building materials. Characters are like sand: infinitely flexible, but you need an enormous amount to build anything. Whole words are like pre-built walls: fast to assemble, but you need a different wall for every situation. Subword tokens are like standard Lego bricks, with a few hundred thousand shapes in the box. Common structures use a single big brick, and unusual structures get assembled from small ones.

The consequence is that the model's "alphabet" is the brick set, not the letters. When you ask how many times the letter r appears in a word, you are asking about the inside of a brick that the model only handles as a whole.

Tokens by example: words, subwords, and bytes

The OpenAI tiktoken library is an open-source tokenizer used with GPT models. Its documentation gives a small example of how it splits text: the word "encoding" can come out as two tokens, "encod" and "ing." Neither piece is a word on its own. The split is simply what the learned vocabulary dictates.

The same documentation gives a useful rule of thumb for English prose: on average, each token corresponds to about 4 bytes of text. For plain English, where one character is usually one byte, that is close to four characters, or roughly three-quarters of a word. Meta's technical report for Llama 3 puts a specific number on a specific tokenizer: its new tokenizer reached 3.94 characters per token on a sample of English data.

What a token is not

A token is not a syllable, a morpheme, or a dictionary word, even though many tokens happen to look like one. It is whatever chunk was statistically convenient when the tokenizer's vocabulary was built. Spaces matter too. In many tokenizers a leading space is part of the token, so "cat" at the start of a sentence and " cat" in the middle of one are two different tokens with two different IDs.

Capitalization, punctuation, and digits also change the split. "Apple," "apple," and " apple" can all map to different tokens. This is why the same sentence in a different case can cost a different number of tokens, and why the text you type and the text the model processes are not quite the same thing.

Byte-pair encoding step by step

Byte-pair encoding (BPE) started as a data-compression trick. According to a 1994 article by Philip Gage in the C Users Journal, the original method scanned a block of data for the most frequent pair of adjacent bytes and replaced every occurrence with a single unused byte, recording the swap in a table. Sennrich and colleagues adapted the idea in 2016 to build vocabularies for neural machine translation, and it is now the basis of the tokenizers behind GPT-style models.

The training procedure is short. The tokenizer never "learns" meaning. It only counts.

1. Collect the words in a large text corpus and count how often each appears.
2. Start with a base vocabulary made of single characters (or bytes).
3. Split every word into those single symbols.
4. Find the most frequent pair of adjacent symbols across the corpus, merge it into a new symbol, and add it to the vocabulary.
5. Repeat step 4 until the vocabulary reaches the target size.

A worked example you can check by hand

The Hugging Face course uses a tiny corpus that is easy to verify: the word "hug" appears 10 times, "pug" 5 times, "pun" 12 times, "bun" 4 times, and "hugs" 5 times. The base vocabulary is the seven letters b, g, h, n, p, s, and u.

Merge	Pair merged	Count	Where the count comes from	New token
1	u + g	20	hug (10) + pug (5) + hugs (5)	ug
2	u + n	16	pun (12) + bun (4)	un
3	h + ug	15	hug (10) + hugs (5)	hug

Each merge picks the most frequent adjacent pair currently in the corpus. Source: Hugging Face LLM Course, chapter 6.

After three merges the vocabulary has ten entries: the seven letters plus ug, un, and hug. The word "hugs" is now two tokens, "hug" and "s." The word "pug" is "p" and "ug."

The order matters. The merges are stored as an ordered list, and at encoding time the tokenizer applies them in exactly the order they were learned. A new word like "bug" becomes "b" plus "ug" because the u-g merge fires, while the "h + ug" merge never applies since there is no h. In the Hugging Face example, a word containing a letter missing from the base vocabulary, such as "mug," would produce an unknown-token marker for the "m."

Why merges, not a dictionary

The approach gives you a vocabulary with a natural ranking. Frequent patterns get long tokens, and rare patterns stay split into short ones. You also get a built-in fallback: if a word is unfamiliar, the tokenizer can always fall back on smaller pieces. The Raschka tutorial on BPE shows this on the sentence "Jack embraced beauty through art and life," where a small vocabulary split "embraced" into four pieces ("em," "br," "ac," "ed"), while "Jack" stayed whole.

A real vocabulary is trained on a corpus billions of words large, so ordinary words end up as single tokens. The small example only shows the mechanism.

Bytes, WordPiece, and SentencePiece: how tokenizers differ

BPE is the common core, but tokenizers differ in what they start from, how they score candidates, and how they treat spaces. The differences show up in practice, mostly in how well a tokenizer handles unusual input.

Byte-level BPE: nothing is unknown

Standard BPE starts from characters, and Unicode has far too many characters to include them all. GPT-2 and RoBERTa solved this by starting from bytes instead. The base vocabulary is just the 256 possible byte values, so any text in any language, including emoji, can be represented. According to the Hugging Face course, this variant means characters are never replaced by an unknown-token marker.

The GPT-2 vocabulary has 50,257 tokens. The OpenAI tiktoken documentation names two later encodings, cl100k_base, described as the GPT-4 encoder, and o200k_base. Raschka's BPE tutorial gives their sizes as 100,256 and 199,997 tokens, so the vocabulary roughly doubled between GPT-4 and GPT-4o.

WordPiece: a different scoring rule

Google developed WordPiece for BERT. It trains like BPE, but instead of merging the most frequent pair, it scores each pair by dividing the pair's frequency by the product of the frequencies of its two parts. The effect is to favor merging pairs whose pieces are individually rare, rather than pairs that are common because their pieces are common. WordPiece marks word-internal pieces with a "##" prefix, and at encoding time it greedily takes the longest vocabulary match from the start of the word. The Hugging Face course notes that Google has not published the training code, so public descriptions are reconstructions.

Unigram and SentencePiece: start big, prune down

The Unigram approach works in reverse. It begins with a very large candidate vocabulary and repeatedly removes the tokens whose absence would increase the corpus loss the least, dropping roughly 10 to 20 percent at each round, until the target size is reached. It tokenizes a word by choosing the most probable segmentation.

SentencePiece, introduced by Taku Kudo and John Richardson at EMNLP 2018, is a toolkit rather than a single algorithm. Its main contribution is that it trains directly from raw sentences and treats whitespace as an ordinary character, written with the symbol "▁." That makes it language-independent: it does not need a separate step to split words first, which matters for languages such as Japanese and Chinese that do not put spaces between words. The Hugging Face course lists T5, ALBERT, mBART, and XLNet among the models that use SentencePiece with Unigram.

Tokenizer family	Starts from	How merges or pieces are chosen	Notable users
Byte-level BPE	256 byte values	Most frequent adjacent pair	GPT-2, RoBERTa, tiktoken encodings
WordPiece	Characters	Pair frequency divided by product of part frequencies	BERT and relatives
Unigram (via SentencePiece)	Large candidate set	Prune pieces that raise corpus loss least	T5, ALBERT, mBART, XLNet

All three produce subword vocabularies, but they disagree on how to build one. Sources: Hugging Face LLM Course, chapter 6; Kudo and Richardson (2018).

For how these tokens feed the rest of a model, see the explainers on how large language models work and transformers explained simply.

Why "strawberry" and arithmetic go wrong

Take the letter-counting failure first. When a model receives the word "strawberry," it receives a short sequence of token IDs. Each ID is turned into a vector of numbers (an embedding) that carries what the model learned about that chunk from billions of examples. Whether the chunk contains zero, one, or two r's is not something the model is handed. It has to have learned the spelling of each chunk indirectly, from text that happens to discuss or spell words out.

The research literature agrees that tokenization is part of the story but does not agree that it is the whole story. A 2025 EMNLP paper by Cosma, Ruseti, Radoi, and Dascalu, "The Strawberry Problem," attributes letter-level failures to tokenization and frames them as a problem of low mutual information between token IDs and character content. In 19 synthetic tasks designed to isolate character-level reasoning, they found that these abilities emerge abruptly and only late in training.

A separate study by Fu and colleagues (arXiv, December 2024) is more cautious. They ran experiments across several models and found that the models can recognize the letters but fail to count them. Word and token frequency in the training data did not significantly affect errors. The strongest correlation was with the number of letters that appear more than once: most models could not reliably count words in which a letter occurs more than twice. In other words, the counting step itself is hard, not only the visibility of the letters.

Spelling out helps

A practical consequence follows from both results. If you ask a model to write out the word letter by letter first, each letter becomes its own token, and the counting problem becomes a much more ordinary one. That is a workaround and not a cure, and the fact that it helps is consistent with the tokenization explanation, though not proof of it.

Arithmetic: where the digits are cut

Numbers get chopped up like any other text, and the cut points affect how well a model adds. Aaditya Singh and DJ Strouse (2024) studied this in "Tokenization counts." The GPT tokenizers they examined have tokens for all 1-, 2-, and 3-digit strings, and by default they split a long number into 3-digit chunks from left to right. The trouble is that long numbers are added from the right, and the left-to-right chunking can misalign the pieces of the two addends with the pieces of the answer.

Their test was simple: insert commas so that the numbers are grouped from the right. On March 2023 models, accuracy on the addition task rose from 75.6 percent to 97.8 percent for GPT-3.5 and from 84.4 percent to 98.9 percent for GPT-4. The authors also noted that Llama and PaLM use single-digit tokenization and that the field has not settled on one approach. The results concern those specific models, and the gap between the two directions narrowed as models got larger.

This is a case where a change to the input, not the model, produced the gain. The model was the same in both conditions. Only the cut points changed.

Why some languages cost more than English

Tokenizer vocabularies are learned from a corpus, and most training corpora contain far more English than anything else. A vocabulary built from that corpus will have long, efficient tokens for common English words and shorter, less efficient pieces for text in underrepresented languages. The same sentence, translated, needs more tokens.

Aleksandar Petrov and colleagues measured this in "Language Model Tokenizers Introduce Unfairness Between Languages," presented at NeurIPS 2023. They found that identical content translated into different languages can produce very different token counts, with differences reaching about 15 times in some cases. The disparities persist even for tokenizers designed for multilingual use, and character-level and byte-level models show gaps of more than 4 times for some language pairs.

Why a token count is a price tag

Token count matters for three practical reasons. Commercial language-model APIs typically charge per token, so a text that tokenizes into more pieces costs more to process. A longer token sequence also takes longer to process, which means higher latency. And it uses more of the model's context window, the fixed number of tokens it can consider at once, leaving less room for the rest of a conversation.

Petrov and colleagues put the first of these plainly: the disparity leads to unfair treatment of some language communities in the cost of accessing commercial language services. To make the scale concrete, take a hypothetical request that needs 100 tokens in English. At the 15-times extreme the same content would need 1,500 tokens, and the bill and the latency would grow by the same factor, other things equal. Most pairs of languages fall well short of that extreme, but the direction of the effect is consistent.

A real attempt to narrow the gap

Tokenizer designers have worked on the problem. Meta's Llama 3 technical report describes a tokenizer with a 128K-token vocabulary that combines 100K tokens from tiktoken with 28K additional tokens added to better support non-English languages. The report states that the new tokenizer improves compression on a sample of English data from 3.17 to 3.94 characters per token relative to the Llama 2 tokenizer, and that the additional tokens helped compression and downstream performance without hurting English.

The arithmetic of that gain is worth working out. If the same text needs 3.17 characters per token before and 3.94 after, the token count for that text falls to about 80 percent of what it was, a reduction of roughly one-fifth.

Glitch tokens: when the vocabulary and the model drift apart

A tokenizer and a language model are trained separately, often on different data. The tokenizer decides which strings deserve their own tokens; the model then learns what each token means. If a string was common enough in the tokenizer's corpus to earn a dedicated token but rare in the model's training data, the model never learns much about it.

Sander Land and Max Bartolo analyzed this in "Fishing for Magikarp," which won an Outstanding Paper Award at EMNLP 2024. They call the result "glitch tokens" or under-trained tokens: entries in the vocabulary that are nearly or entirely absent from the model's training data. Because tokenizer creation and model training are disconnected, specific inputs such as the string "SolidGoldMagikarp" can trigger unwanted behavior. The authors report that such tokens are common across a range of models and propose automatic ways to detect them, combining tokenizer analysis, model-weight indicators, and prompting.

The phenomenon is a good reminder of what a token really is. It is an index into a table of learned vectors. If a row of that table was barely trained, the model has nothing meaningful to retrieve, and its behavior around that token can be erratic. This is one concrete way that tokenization problems become hallucination and reliability problems.

Vocabulary size: the trade-off every tokenizer makes

Choosing the vocabulary size is a bargain with three parties: the sequence length, the model's size, and the tokenizer's coverage.

A larger vocabulary means longer tokens on average, so a given text becomes a shorter sequence. A shorter sequence means cheaper attention, because the number of position-to-position comparisons grows with the square of the length. Using the Llama 3 figures, the token count drops to about 0.80 of its previous value, and the square of 0.80 is about 0.65, so the length-dependent part of the attention cost for that text falls by roughly a third, other things equal.

The cost on the other side is the size of the embedding table and the output layer. The model needs one learned vector for each vocabulary entry, and on every step it must produce a score for each possible next token. As a hypothetical example, a vocabulary of 128,000 tokens and vectors of 4,096 numbers each implies an embedding table of about 524 million numbers. That is a meaningful slice of a model with a few billion parameters. Rare tokens in a large vocabulary also receive few training examples, which makes them more likely to end up under-trained, the problem described above.

Choice	Benefit	Cost
Smaller vocabulary	Smaller embedding and output tables; every token seen often	Longer sequences; more attention compute; worse for non-English text
Larger vocabulary	Shorter sequences; better compression, especially with added multilingual tokens	Bigger embedding table; more rarely trained tokens
Byte-level base	No unknown characters; any language or emoji representable	Long sequences when few merges apply

No size is free; each tokenizer picks a point on the curve.

Why not just use characters or bytes?

Researchers have tried models that operate directly on bytes or characters and skip learned vocabularies. Petrov and colleagues found that even such models show gaps of more than 4 times between some language pairs. Tokenization is not the only source of unevenness, then, though a poor tokenizer can make it worse. Byte-level models also face the long-sequence problem in its purest form, which is why subword vocabularies remain the default.

What this means when you use a chatbot

You cannot see tokens in the chat window, but their effects are visible if you know where to look. Here are the practical patterns that follow from how tokenizers work.

• Letter-level tasks are unreliable. Counting letters, reversing words, and checking anagrams all require access to characters the model does not directly see. Writing the word out letter by letter first improves the odds.
• Long numbers are risky. Digit grouping affects addition and multiplication, so results on large numbers deserve a check with a calculator.
• Non-English text may cost more and fit less. If you use an API, expect the same content to consume more of the budget in some languages, and see the explainer on prompt caching for one way costs are managed.
• Odd strings can behave oddly. Unusual usernames, rare compound tokens, and long runs of repeated characters sometimes trigger unexpected output.
• Sampling happens over tokens. When a model "chooses" its next word, it is choosing the next token, as described in the guide to temperature and top-p sampling, and a word split into three tokens requires three consecutive choices.

See your own tokens

You do not have to take any of this on trust, because tokenizers are inspectable. The open-source tiktoken library turns a string into a list of integers with a couple of lines of Python, and the same list can be decoded token by token to show where the cuts fall. Running one sentence through two encodings, for example the GPT-4-era cl100k_base and the larger o200k_base, is a quick experiment: the two vocabularies differ in size (100,256 against 199,997 entries, per Raschka's count), so the same text will generally split differently and the token counts will not match.

The same habit helps with budgeting. The "four characters per token" rule is an average over English prose, and it can be well off for code, for numbers, and for text in other languages. If a cost or a context limit matters, count tokens with the tokenizer that the specific model uses rather than estimating from word counts. A tokenizer built for one model is not interchangeable with another's, since the vocabularies and the merge rules differ, and an ID that means one chunk in one model can mean something else entirely in another.

None of the patterns above are bugs in a particular product. They follow from the interface between text and numbers, and every current model has a version of them. For the larger map of how these pieces fit together, the Complete AI Guide covers the stack from tokenization through training and deployment, and how AI models are trained covers where the tokenizer sits in the pipeline.

The takeaway

Tokenization is the first decision in every language model, and it is made before the model ever sees a word. Byte-pair encoding builds a vocabulary by counting which neighboring pieces occur together most often and merging them, which gives common words single tokens and rare words several. The side effects are well documented: letter counting is hard because letters are hidden inside tokens, arithmetic depends on where digits are cut, some languages need up to about 15 times as many tokens as others for the same content, and vocabulary entries that the model rarely saw can behave strangely. None of that means the models are not intelligent. It means they work on a different alphabet than the one you type, and a good deal of apparent AI clumsiness is the cost of that translation layer.

Frequently asked questions

Q: What is a token in a large language model? A: A token is a chunk of text, from a single character to a whole common word, that appears in a fixed vocabulary learned in advance. The model reads and writes sequences of token IDs. For English, a token averages about four bytes, which is roughly three-quarters of a word, though the split depends on the tokenizer.

Q: What is byte-pair encoding? A: Byte-pair encoding is an algorithm that builds a token vocabulary by starting from single characters or bytes and repeatedly merging the most frequent adjacent pair into a new token. It began as a compression method in 1994 and was adapted for neural translation by Sennrich, Haddow, and Birch in 2016.

Q: Why can't ChatGPT count the letters in a word? A: The model receives token IDs, not letters, so the spelling inside each token is only learned indirectly. Research also finds that counting is hard for models even when they can recognize the letters, especially when a letter repeats more than twice. Spelling the word out letter by letter first usually helps.

Q: Why do AI models struggle with long arithmetic? A: Numbers are split into digit chunks, and the GPT tokenizers studied by Singh and Strouse cut them into 3-digit groups from left to right, which misaligns with how addition works. Grouping digits from the right with commas raised GPT-4 addition accuracy from 84.4 percent to 98.9 percent in their March 2023 test.

Q: Why do some languages cost more to process than English? A: Tokenizer vocabularies are learned mostly from English-heavy text, so other languages are split into more, smaller pieces. Petrov and colleagues found token-count differences of up to about 15 times for the same content, which raises per-token API cost and latency and shrinks the usable context window.

Q: What is a glitch token? A: A glitch token is a vocabulary entry that the model saw rarely or never during training, so it has no meaningful learned representation. Land and Bartolo (EMNLP 2024) show such tokens appear across many models and can trigger unwanted behavior, with "SolidGoldMagikarp" as the best-known example.

Q: Is a bigger vocabulary always better? A: No. A bigger vocabulary shortens sequences, which reduces attention cost, but it enlarges the embedding table and leaves more tokens with few training examples. Llama 3 moved to a 128K vocabulary and gained about 24 percent more characters per token on English text, but each model weighs the trade-off differently.

Sources

• Neural Machine Translation of Rare Words with Subword Units (Sennrich, Haddow, Birch; arXiv:1508.07909): the 2016 ACL paper that brought BPE to neural language processing.
• SentencePiece (Kudo and Richardson; arXiv:1808.06226): language-independent subword tokenizer, EMNLP 2018 demo paper.
• Hugging Face LLM Course, Byte-Pair Encoding tokenization: the hug/pug/pun/bun worked example and byte-level BPE.
• Hugging Face LLM Course, WordPiece tokenization: scoring rule and the ## prefix.
• Hugging Face LLM Course, Unigram tokenization: the prune-down approach and SentencePiece usage.
• openai/tiktoken on GitHub: encodings and the roughly 4 bytes per token average.
• Language Model Tokenizers Introduce Unfairness Between Languages (Petrov et al.; arXiv:2305.15425): token-count disparities of up to about 15 times.
• Tokenization counts (Singh and Strouse; arXiv:2402.14903): digit chunking and arithmetic accuracy.
• Why Do LLMs Struggle to Count Letters? (Fu et al.; arXiv:2412.18626): correlational study of letter-counting errors.
• The Strawberry Problem (Cosma et al.; EMNLP 2025): character-level understanding in tokenized models.
• Fishing for Magikarp (Land and Bartolo; EMNLP 2024): under-trained glitch tokens.
• The Llama 3 Herd of Models (arXiv:2407.21783): the 128K vocabulary and compression figures.
• Sebastian Raschka, BPE from scratch: vocabulary sizes of GPT-2, GPT-4, and GPT-4o encodings.