Natural Language Processing: How Computers Understand Text

Photo: Google DeepMind / Pexels
How do computers understand text when all they can do is arithmetic? They do not read the way people do. A natural language processing (NLP) system first chops text into pieces, converts each piece into a list of numbers, and then lets a neural network adjust those numbers according to the surrounding words. Everything a language tool does, from translating a menu to flagging an angry review, rests on that conversion. This article walks the pipeline stage by stage, with small worked examples you can check by hand.
The core problem: language is not numbers
"I saw the man with the telescope" has at least two readings. Either you used a telescope to see him, or he was holding one. A person resolves it from the situation in an instant. A program has only the string of characters, and nothing in those characters says which reading is intended.
Computers operate on numbers, and language is a sequence of symbols whose meaning depends on syntax, word choice, and the world outside the text. The central job of NLP is to represent language numerically in a way that preserves enough meaning for useful work. Most of the field's history is a series of better answers to that single problem, a story that sits inside the larger history of artificial intelligence.
Here is the modern pipeline in one table. Each row gets its own section below.
| Stage | What happens | Example |
|---|---|---|
| Tokenization | Text is split into subword pieces | "unhelpfulness" becomes un, helpful, ness |
| Embedding | Each token becomes a vector of numbers | cat becomes [0.8, 0.6, 0.1] (illustrative) |
| Contextualization | A neural network updates each vector using its neighbors | "bank" gets different vectors in two sentences |
| Task head | A small output layer turns vectors into an answer | A label such as positive or negative |
| Decision | Software acts on the answer and its confidence | Route to a human if confidence is low |
Step one: tokenization
Before any math, text must be broken into tokens, the units the model reads. The naive approach splits on spaces and punctuation, so "Hello, world!" becomes four pieces: Hello, a comma, world, and an exclamation mark. That works for tidy English and fails quickly on real text.
Contractions raise the question of whether "don't" is one token, two, or three. Hashtags, URLs, and compound words have no obvious boundaries. Chinese and Japanese are written without spaces between words. And any fixed word list will meet words it has never seen, such as a new brand name or a typo, with no way to represent them.
Subword tokenization with a worked example
The fix is to tokenize into subwords, pieces smaller than words but larger than letters. The most common method is byte pair encoding (BPE), which Rico Sennrich, Barry Haddow, and Alexandra Birch applied to translation in a 2015 paper. Their idea was to represent rare and unknown words as sequences of subword units, which makes the vocabulary open-ended. In their tests, subword models improved English-German translation by 1.1 BLEU and English-Russian by 1.3 BLEU over a back-off dictionary baseline.
The algorithm is simple enough to run on paper. Take an invented training vocabulary in which each word is split into characters and has a count: "low" appears 5 times, "lower" 2 times, "newest" 6 times, and "widest" 3 times.
- Count every adjacent pair of symbols. The pair "e" followed by "s" appears 9 times (6 in "newest" and 3 in "widest"), which ties with "s" followed by "t". Break the tie by picking "e s".
- Merge it into a new symbol "es". Now the most frequent pair is "es" followed by "t", also 9 times.
- Merge that into "est".
- Keep merging the most frequent pair until the vocabulary reaches a target size.
After a few steps, "est" is a single token. A word the model has never seen, such as "tallest," can still be written as "tall" plus "est." Common words end up as single tokens, and rare words break into familiar pieces. For a deeper look, see our explainer on how LLM tokenization works.
BERT uses a close relative called WordPiece with a vocabulary of 30,000 tokens. A commonly cited rule of thumb for English is about four characters, or three quarters of a word, per token. The ratio depends on the tokenizer and the language.
Tokenization is not equally fair to every language
A tokenizer trained mostly on English splits English efficiently and other languages less so. A NeurIPS 2023 paper by Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi found that the same text translated into different languages can differ in tokenized length by up to 15 times in some cases.
That matters in two ways. Models with a fixed context window, the maximum amount of text they can consider at once, effectively see less of a document in a token-hungry language. And when usage is billed per token, the same message can cost more in one language than another. Our explainer on the context window covers the first effect.
Step two: turning tokens into vectors
Once text is tokens, each token needs a numeric form. The simplest option is one-hot encoding: with a vocabulary of 50,000 tokens, each token becomes a list of 50,000 numbers, all zeros except a single 1. This works mechanically but says nothing about meaning. In that scheme, "cat" is exactly as different from "kitten" as it is from "democracy."
An embedding replaces the long sparse list with a short dense one, typically a few hundred numbers, arranged so that tokens used in similar ways get similar numbers.
Measuring closeness with cosine similarity
The standard way to compare two vectors is cosine similarity, which measures the angle between them. A value near 1 means they point the same way, and a value near 0 means they are unrelated. Here is an illustrative example with three-number vectors; real embeddings have hundreds of numbers, but the arithmetic is the same.
| Word | Vector (invented for illustration) |
|---|---|
| cat | [0.8, 0.6, 0.1] |
| kitten | [0.7, 0.7, 0.2] |
| democracy | [0.1, 0.2, 0.9] |
For cat and kitten, the dot product is 0.8 x 0.7 + 0.6 x 0.7 + 0.1 x 0.2 = 1.00. The lengths are about 1.005 and 1.010, so the cosine similarity is roughly 0.98. For cat and democracy, the dot product is 0.08 + 0.12 + 0.09 = 0.29, and the cosine similarity is roughly 0.31. The numbers say "cat" is far closer to "kitten" than to "democracy," which is the property that embeddings are trained to produce. Our guide to embeddings and vector search and our primer on vector databases show how this comparison powers search.
Word2Vec: learning meaning from neighbors
Where do the numbers come from? Nobody writes them. The key idea, usually traced to the linguist J. R. Firth in 1957, is that a word is known by the company it keeps. Word2Vec, introduced in a paper first posted in January 2013 by Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, turned that idea into a fast training method. The authors reported that it takes "less than a day to learn high quality word vectors from a 1.6 billion words data set."
The method trains a shallow network on a prediction game with two variants. In CBOW (continuous bag of words), the network guesses a word from its neighbors. In skip-gram, it guesses the neighbors from a word. Consider "the cat sat on the mat" with a window of two words on each side. For the center word "sat," skip-gram creates training pairs like (sat, the), (sat, cat), (sat, on), and (sat, the). The network is rewarded for predicting those neighbors, and as a side effect its internal weights become the word's vector.
Nobody cares about winning the game. The vectors are the prize.
Geometry that encodes relationships
The vectors turned out to hold regularities. In a 2013 NAACL paper, Mikolov, Wen-tau Yih, and Geoffrey Zweig showed that relationships appear as consistent offsets between vectors: king minus man plus woman lands near queen. Their vectors answered syntactic analogy questions correctly almost 40% of the time. No one programmed the relationship. It emerged because the words in each pair appear in parallel patterns of context.
The limit of this approach is that each word gets exactly one vector. "Bank" has to blend the riverbank and the savings bank into a single point, so the representation is only as good as the average meaning.
Step three: making meaning depend on context
The next advance made a word's vector a function of the whole sentence. In February 2018, Matthew Peters and colleagues introduced deep contextualized word representations, where each word's vector is "a learned function of the internal states of a deep bidirectional language model." Adding them to existing systems improved the state of the art on six tasks, including question answering and sentiment analysis. The method is known as ELMo.
BERT: how the pretraining game works
In October 2018, Jacob Devlin and colleagues at Google posted BERT, which stands for Bidirectional Encoder Representations from Transformers. It swaps the older recurrent networks for the Transformer architecture; see our explainer on how Transformers pay attention to words. A Transformer lets every token look at every other token in the input and decide which ones matter for its own meaning.
BERT learns by playing two games on unlabeled text. Think of the first as a fill-in-the-blank exercise.
Masked language modeling. The training code chooses 15% of the token positions at random. Of those, 80% are replaced with a special [MASK] symbol, 10% are replaced with a random token, and 10% are left alone. The model must predict the original token at every chosen position. "The [MASK] sat on the mat" should come out as "cat." Using both the left and right context is what makes the model bidirectional, and the 80/10/10 split exists because the [MASK] symbol never appears in real text at fine-tuning time.
Next sentence prediction. The model sees two sentences, and half the time the second really followed the first in the source text. It predicts which case it is, which is meant to teach relationships between sentences. The paper reported that the final model reached 97% to 98% accuracy on this task.
The input to the network is built by adding up three embeddings for each token: the token itself, which sentence it belongs to, and its position in the sequence. Two special tokens frame the input: [CLS] at the start, whose final vector serves as a summary of the whole sequence, and [SEP] between sentences.
| Model | Layers | Hidden size | Parameters | Pretraining data |
|---|---|---|---|---|
| BERT-base | 12 | 768 | 110 million | BooksCorpus (800M words) plus English Wikipedia (2,500M words) |
| BERT-large | 24 | 1,024 | 340 million | Same |
Source: Devlin et al. (2018), section 3.
The paper notes that BERT-base was chosen to match the size of OpenAI's GPT for comparison. The difference lies in attention. BERT uses bidirectional self-attention, while GPT uses a constrained form in which each token can attend only to context on its left. That left-to-right design suits generating text one word at a time, and the bidirectional design suits analyzing a complete passage. The paper's footnote names the two families: a "Transformer encoder" for the bidirectional version and a "Transformer decoder" for the left-only version.
A follow-up study, RoBERTa, concluded in 2019 that "BERT was significantly undertrained," and showed that a better-tuned version could match models published later. Training choices, not only architecture, decide how good a model gets.
Step four: fine-tuning one model for many jobs
Before BERT, each language task usually meant a separate model built for that task. After BERT, the standard recipe was to take a pretrained model and fine-tune it, meaning to continue training briefly on a small labeled dataset for the target task. Our explainer on transfer learning describes the general idea, and the guide to fine-tuning LLMs covers its modern form.
Fine-tuning is cheap compared with pretraining. The BERT paper states that all of its fine-tuning results can be replicated in at most one hour on a single Cloud TPU, or a few hours on a GPU, starting from the same pretrained model. As an example, a SQuAD question-answering model can be trained in about 30 minutes on a single Cloud TPU to reach a development F1 score of 91.0.
What changes between tasks is only the final layer. According to the paper, for token-level tasks such as tagging and question answering, the output vectors for each token feed an output layer. For classification tasks such as sentiment analysis, the [CLS] vector feeds an output layer instead.
| Task | What goes in | What the output layer predicts |
|---|---|---|
| Sentiment classification | One text | One label from the [CLS] vector |
| Entailment or paraphrase | Two sentences packed together | One label for the pair |
| Named entity recognition | One text | A label for every token |
| Question answering | A question and a passage | Start and end positions of the answer span |
A pipeline in practice
Consider a support team that wants to sort incoming tickets into categories. This example is illustrative, not drawn from any particular company.
- Tokenize each ticket with the model's tokenizer.
- Pass the tokens through a pretrained transformer to get contextual vectors.
- Fine-tune on a few thousand tickets that people have already labeled.
- At run time, the model outputs a category and a confidence score.
- Tickets above a chosen confidence threshold are routed automatically, and the rest go to a person.
The last step is the part that keeps the system honest. A classifier always outputs something, even for a ticket unlike anything it was trained on, so a confidence cutoff and human review are how teams catch the model's weak spots. Our article on LLM evaluation covers how to measure whether such a system works.
Sentiment analysis: reading tone in text
Sentiment analysis labels text as positive, negative, or neutral. It is one of the most commercially common NLP tasks, used to scan reviews and social posts at a scale no team could read.
The simple approach is a lexicon: count positive words and negative words and take the difference. It fails on negation ("not bad"), on mixed statements ("the acting was terrible, but the story was great"), and on context that flips a word's polarity ("good riddance"). A 2002 paper by Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan tested movie-review classification and found that Naive Bayes, maximum entropy, and support vector machines definitively outperformed human-produced baselines, meaning word lists chosen by people. The same study found these methods did worse on sentiment than on ordinary topic classification, a sign that tone is harder than subject matter.
Fine-tuned transformers go further because the contextual vector for "bad" already depends on whether "not" precedes it. Three problems remain. A model trained on movie reviews may stumble on restaurant reviews, which is called domain shift. Implied sentiment, as in "the food was slow," carries no obvious sentiment word. And sarcasm, where the literal and intended meanings oppose each other, depends on shared context that a short text may not contain.
In finance, sentiment models are applied to news and earnings transcripts, with caveats we cover in sentiment analysis for stocks. Because these models learn from human-written text, they can inherit human biases, a topic our article on AI bias and discrimination develops.
Named entity recognition and information extraction
Named entity recognition (NER) finds and labels mentions of people, organizations, places, dates, and amounts. In "Apple CEO Tim Cook announced record profits on Tuesday," a tagger should mark Apple as an organization, Tim Cook as a person, and Tuesday as a date.
The hard part is ambiguity. "Apple" could be a fruit, "Jordan" a country or a person, and only the surrounding words decide. A contextual model has an advantage here, because its vector for "Apple" differs between "Apple shares fell" and "she ate an apple."
NER feeds information extraction, which turns unstructured text into structured records. A set of research abstracts becomes a table of studies and findings. Clinical notes become searchable fields. The output is only as reliable as the tagger, so high-stakes uses add human review.
Machine translation: the same pipeline, two languages
Translation shows the pipeline at full scale. The early approach used hand-written rules, which were rigid. Statistical systems then learned from aligned pairs of documents in two languages. The neural approach, built on an encoder that reads the source and a decoder that writes the target, came next.
An early neural design squeezed the whole source sentence into one vector. In a paper first posted in September 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio argued that the fixed-length vector is a bottleneck and let the decoder search for the source words relevant to each target word. That is the attention idea: the translator glances back at the source rather than working from memory alone.
The 2017 paper Attention Is All You Need made attention the main computation. It reported 28.4 BLEU on English-to-German and 41.8 BLEU on English-to-French on the WMT 2014 test sets, the latter after training for 3.5 days on eight GPUs. In 2016, Google researchers reported that their neural system cut translation errors by an average of 60% relative to the earlier phrase-based system in a human evaluation on isolated simple sentences. Simple sentences are an easier test than full documents, so the figure should be read narrowly.
Coverage has grown with the models. In June 2024, Google added 110 languages to Google Translate using its PaLM 2 language model, calling it the largest expansion ever, covering more than 614 million speakers. For a practical comparison of translation products, see our guide to AI translation tools.
How we check whether a system "understands"
Passing tests is not the same as understanding, but tests are what we have. The standard approach is a benchmark: a fixed set of questions with known answers. BERT's results on GLUE, a bundle of language-understanding tasks, set the pattern. BERT-large scored 80.5 on the GLUE leaderboard, 7.7 points above the previous best, and 93.2 F1 on SQuAD v1.1.
Benchmarks have a lifespan. A 2019 paper introducing the harder SuperGLUE benchmark said it was needed because performance on GLUE had surpassed the level of non-expert humans. Models can also fail in ways a benchmark never probes. They can state falsehoods fluently, a failure covered in our explainer on why AI hallucinates facts, and they can be thrown off by small changes in wording.
None of this makes the technology fake. It means scores describe performance on a particular test, and a reader should ask what the test was.
Key takeaways
- Tokenization converts raw text into discrete units, and modern systems use subword methods such as BPE so that rare words break into familiar pieces.
- Tokenization is uneven across languages, with token counts for the same text differing by up to 15 times in some pairs.
- Embeddings place tokens in a vector space where similar usage means small angles, and cosine similarity measures the closeness.
- Word2Vec learns vectors by predicting neighbors, and its vectors encode relationships such as king minus man plus woman.
- BERT made vectors depend on context by masking 15% of tokens and predicting them from both sides.
- Pretraining plus fine-tuning lets one model be adapted to sentiment, entity recognition, question answering, and translation by changing only the final layer.
- Scores on benchmarks are snapshots of performance on a specific test, not proof of understanding.
Frequently asked questions
Q: What is the difference between NLP and large language models? A: NLP is the broad field of getting computers to process human language. Large language models are one family of NLP systems, trained to predict text at very large scale. NLP also includes narrower tools for tagging, translation, entity recognition, and classification.
Q: How does tokenization affect model performance? A: Tokenization decides what units the model sees. A tokenizer trained mostly on English may split text in other languages into many more pieces. A 2023 study found differences of up to 15 times in token counts for the same text, which shrinks the effective context window and raises per-token costs.
Q: Why is the king minus man plus woman example important? A: It showed that relationships between concepts could be encoded as directions in a vector space without anyone programming them. The 2013 paper behind it reported that learned vectors answered syntactic analogy questions correctly almost 40% of the time.
Q: What is fine-tuning in BERT? A: Fine-tuning continues training a pretrained model on a small labeled dataset for one task. The BERT paper says its fine-tuning results can be replicated in at most one hour on a single Cloud TPU, starting from the same pretrained model.
Q: How do computers handle sarcasm? A: Imperfectly. Sarcasm depends on the gap between literal and intended meaning, which often relies on context outside the text, such as the speaker's relationship to the listener. Contextual models do better than word counts, but short messages often lack the clues needed.
Q: Is NLP solved? A: No. Systems perform well on many benchmarks, but they still state falsehoods, struggle with domain shift, and serve some languages much better than others. Whether predicting text amounts to understanding it is still debated.
Read next:
- How Large Language Models Work: Transformers, Attention, and Why They Hallucinate
- Edge AI: How AI Runs on Your Phone
- Generative Adversarial Networks: How GANs Work
Sources
- Sennrich, Haddow, and Birch (2015). Neural Machine Translation of Rare Words with Subword Units
- Petrov et al. (2023). Language Model Tokenizers Introduce Unfairness Between Languages
- Mikolov et al. (2013). Efficient Estimation of Word Representations in Vector Space
- Mikolov, Yih, and Zweig (2013). Linguistic Regularities in Continuous Space Word Representations
- Peters et al. (2018). Deep contextualized word representations
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Liu et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Pang, Lee, and Vaithyanathan (2002). Thumbs up? Sentiment Classification using Machine Learning Techniques
- Bahdanau, Cho, and Bengio (2014). Neural Machine Translation by Jointly Learning to Align and Translate
- Vaswani et al. (2017). Attention Is All You Need
- Wu et al. (2016). Google's Neural Machine Translation System
- Wang et al. (2019). SuperGLUE
- Google (2024). Google Translate adds 110 languages
Follow new articles
Add SciHub101 to your feed reader to see new science and technology stories.
About the Author
SciHub101 is an independent science and technology publication founded and edited by Buğra Sözeri, a software engineer and app developer based in Türkiye. AI tools assist with drafting and research. Source documentation in the existing archive is uneven, and articles are being reviewed and corrected. The Editorial Team byline identifies editorial responsibility; it does not certify specialist or independent human review. Where we compare products, anything built by our own team is labelled as such — see the editorial guidelines and About page for how corrections and disclosures work.
Related Articles

Reinforcement Learning Explained: How AI Learns to Play Games
AlphaGo did not read a Go textbook for its strongest play. It improved by playing itself, guided by win or lose. Here is the math and the history behind how reinforcement learning works.

Generative Adversarial Networks: How GANs Work
Two neural networks play a game. One forges images, the other tries to catch the forgeries, and each failure teaches the forger something. That is a GAN, and its strengths and flaws all follow from that game.

How Image Recognition Works: The Science of Convolutional Neural Networks
In 2010 the best image classifier missed the right label in its top five guesses about 28% of the time. By 2015 that was 3.57%. Here is what changed, and how a CNN actually sees.

How Large Language Models Work: Transformers, Attention, and Why They Hallucinate
Meta trained Llama 3 on more than 15 trillion tokens using 7.7 million H100 GPU-hours. Here is how next-token prediction at that scale turns into a conversation partner.

Edge AI: How AI Runs on Your Phone
Server-side chatbots run on racks of GPUs in data centers. Face ID, voice dictation, and a growing number of language features run on a chip the size of a fingernail in your pocket. The science of shrinking large models to fit is called edge AI.

How Diffusion Models Work: Stable Diffusion, DALL-E, and the Science of AI Art
Type a sentence, get a picture. Behind Stable Diffusion and DALL-E sits one idea borrowed from thermodynamics: destroy images with noise in small steps, train a network to undo each step, then run the process backward from static.