Natural Language Processing: From Rules to Learning

In January 1954, IBM and Georgetown University publicly demonstrated Russian-to-English machine translation. The system knew 250 words and six grammar rules. In March 2025, researchers reported that a language model given a persona prompt was judged to be the human more often than the real people it was compared with, in five-minute chat tests. Between those two points, natural language processing (NLP), the field that teaches computers to work with human language, changed its method three times. Each change fixed the main weakness of the one before it.

The question that set the target

In 1950, Alan Turing published "Computing Machinery and Intelligence" in the journal Mind. He proposed replacing the vague question "can machines think?" with a game. A judge exchanges written messages with a hidden human and a hidden computer. If the judge cannot reliably tell which is which, the computer has done well at the game.

For decades the test worked as a target more than a measurement. In 2025, Cameron Jones and Benjamin Bergen ran a three-party version in which participants chatted for five minutes with a person and a system at the same time, then picked which one was human. In their preprint, GPT-4.5 with a humanlike persona prompt was picked as the human 73% of the time, more often than the real human participants. LLaMa-3.1-405B with the same prompt was picked 56% of the time. The study's baselines were picked far less often: ELIZA, a 1960s program, won 23% of the time, and a GPT-4o baseline won 21%.

Two cautions apply. The result comes from a preprint, and it measures whether a conversation fools a judge in five minutes, not whether the system comprehends what it says. As we will see, the gap between those two things is the oldest argument in the field. For the wider timeline of the discipline, see our history of artificial intelligence.

When linguists wrote the rules (1950s to 1980s)

The earliest NLP systems were built by hand. Researchers took what linguists knew about grammar, wrote it down as rules, and had the computer apply them.

The Georgetown-IBM demonstration in January 1954 is the usual starting point. Historian John Hutchins describes it as a small-scale experiment of just 250 words and six grammar rules. It drew wide public interest and, in Hutchins's account, raised expectations that high-quality automatic translation was near. The expectation was wrong, and it took decades to find out why.

How ELIZA faked a conversation

A better window into rule-based NLP is ELIZA, written by Joseph Weizenbaum at MIT between 1964 and 1967 and published in 1966. ELIZA ran a script called DOCTOR that imitated a Rogerian psychotherapist, a style of therapy in which the therapist mostly reflects the patient's statements back. Weizenbaum chose that role because it let the program avoid needing any knowledge of the real world.

The mechanism was three steps. ELIZA scanned the input for keywords, each with a rank set by the script, and picked the highest-ranked one. A decomposition rule split the sentence into fragments around that keyword. A reassembly rule rearranged the fragments into a reply, often a question. "You are very helpful" could become "What makes you think I am very helpful?" If no keyword appeared, the program fell back on a content-free remark such as "Please go on."

Nothing in that loop understands anything. What surprised Weizenbaum was how people reacted. He was struck that users, including his secretary, attributed human feelings to the program, and he later wrote that a very brief exposure to a simple program could produce "powerful delusional thinking in quite normal people." The tendency to read understanding into fluent output is now called the ELIZA effect, and it returns every time a more capable chatbot appears. Our piece on the evolution of chatbots follows that thread from ELIZA to ChatGPT.

Why rules hit a wall

Rule-based systems failed for a structural reason. Writing a rule requires foreseeing the case it covers, and language keeps producing cases nobody foresaw. A grammar rule such as "a sentence is a noun phrase followed by a verb phrase" works until the first sentence that breaks it. Each fix adds a rule that interacts with all the others, and the system grows brittle.

Imagine trying to write a cookbook that covers every dish anyone will ever cook, by listing each recipe in advance. You can cover the common dishes, but the first person to improvise gets no help. Rule-based NLP worked in narrow settings with controlled vocabulary, such as scripted dialogue, and failed in open text. Our article on expert systems covers the broader family of hand-encoded knowledge that this approach belonged to.

The most consequential judgment on the era came from outside. A committee led by John R. Pierce of Bell Telephone Laboratories, set up by the U.S. government in April 1964, issued the ALPAC report in 1966. Its skeptical assessment of machine translation is widely credited with reducing U.S. funding for the field dramatically and with starting the first AI winter, a period when funding and interest in AI research dried up.

Counting instead of writing: the statistical turn (1990s to 2000s)

The next idea was to stop telling the computer what language looks like and let it measure. Instead of a linguist writing "the is followed by a noun or adjective," a program would count how often words follow other words in a very large collection of real text.

In 1990, an IBM group published A Statistical Approach to Machine Translation in the journal Computational Linguistics, making the case for treating translation as a problem of probabilities rather than hand-written rules. The same logic spread through the field, and for roughly twenty years probabilities were the standard tool.

N-grams with a worked example

An n-gram language model predicts the next word from the previous n minus 1 words. A bigram model (n = 2) looks only at the single previous word, and a trigram model (n = 3) looks at two.

Here is a tiny, invented training set of two sentences: "the cat sat on the mat" and "the cat ate the fish." The word "the" appears four times. It is followed by "cat" twice, "mat" once, and "fish" once. So the model estimates:

• P(cat | the) = 2 / 4 = 0.50
• P(mat | the) = 1 / 4 = 0.25
• P(fish | the) = 1 / 4 = 0.25

Scale that up to billions of words and the counts become a usable predictor. That predictor powered early speech recognition, spelling correction, and translation scoring. Its weakness is also visible in the toy example: the model knows nothing beyond the last word or two.

Hidden Markov models and tagging

Statistical models also tackled labeling tasks. In part-of-speech tagging, the goal is to mark each word as a noun, verb, adjective, and so on. A hidden Markov model treats the tags as an unseen sequence that generates the visible words, and it learns two kinds of probability from labeled text: how likely each tag is to follow another, and how likely each tag is to produce a given word. Given a new sentence, it finds the most probable tag sequence. Later systems used discriminative models such as conditional random fields, which learn directly to separate right labelings from wrong ones.

How good did tagging get? In a 2011 paper, Stanford's Christopher Manning reported that taggers had reached about 97.3% accuracy per token, but only about 56% per sentence. Because one wrong tag spoils a sentence for downstream tools, the sentence-level number is a more honest picture of reliability. Manning also traced the common belief that humans cannot beat 97% to an old note in the Penn Treebank documentation. When two human annotators tagged text from scratch, they disagreed 7.2% of the time, and when they corrected an automatic tagger's output, the disagreement fell to 4.1%, partly because both were adopting the tagger's conventions.

The long-range problem

Counting had a ceiling, and the ceiling was context. Consider two sentences that differ by one word:

• "The trophy didn't fit in the suitcase because it was too big."
• "The trophy didn't fit in the suitcase because it was too small."

In the first, "it" is the trophy. In the second, "it" is the suitcase. Deciding requires knowing how objects and sizes behave, and no count of nearby words supplies that. Short windows cannot hold the dependency, and widening the window makes the counts too sparse to trust, because most long word sequences never appear in any training text.

Era	Where knowledge comes from	Strength	Main limit
Rules (1950s to 1980s)	Linguists write it down	Transparent; precise in narrow domains	Brittle; cannot cover open text
Statistics (1990s to 2000s)	Counts over text and labeled data	Robust; learns from examples	Short context; needs labeled data
Neural embeddings (2013 onward)	Vectors learned from raw text	Captures similarity of meaning	One vector per word, no context
Pretrained transformers (2018 onward)	Large-scale self-supervised training	Context-aware; transfers across tasks	Cost; opaque; invents facts

Core tasks and how each one changed

NLP is a toolkit of tasks, and each task went through the same arc. Here is where several of them stand.

Tokenization is the splitting of text into units a model can handle. Splitting on spaces fails on contractions, hyphens, and languages written without spaces. A 2015 paper by Rico Sennrich, Barry Haddow, and Alexandra Birch, Neural Machine Translation of Rare Words with Subword Units, proposed encoding rare and unknown words as sequences of subword units using byte pair encoding. Against a back-off dictionary baseline, it improved English-German translation by 1.1 BLEU and English-Russian by 1.3 BLEU on the WMT 15 test sets. (BLEU is a score that compares machine output with human reference translations.) Our explainer on how BPE tokenization works walks through the algorithm.

Part-of-speech tagging moved from hidden Markov models to discriminative models to neural networks, with per-token accuracy near the ceiling described above.

Parsing recovers sentence structure. Dependency parsing links words by grammatical relation, so in "Alice ate the apple," "ate" is the root, "Alice" is its subject, "apple" is its object, and "the" attaches to "apple." Constituency parsing builds nested phrases, written as (S (NP Alice) (VP ate (NP the apple))). Both are now typically done with neural networks.

Named entity recognition (NER) finds mentions of people, organizations, places, and dates, and it is a workhorse for building structured data from text. The difficulty is ambiguity: "Apple" can be a company or a fruit, and only context decides.

Coreference resolution links pronouns to the things they refer to, as in the trophy sentences above. It remains one of the places where models still make instructive mistakes.

Word embeddings: meaning as geometry

By the early 2010s, the field had a bottleneck. Statistical systems treated words as arbitrary symbols, so "cat" and "kitten" were as unrelated as "cat" and "tax." Embeddings fixed that by representing each word as a vector, a list of numbers, such that words used in similar ways get similar vectors.

The guiding idea is much older than the technology. In 1957, the linguist J. R. Firth wrote what is usually quoted as "you shall know a word by the company it keeps." If "cat" and "kitten" appear among the same neighbors (feed, purr, vet), their vectors should end up close.

A useful picture is a map. Cities get coordinates, and cities that are near each other in the real world get nearby coordinates. An embedding gives each word coordinates in a space with hundreds of dimensions instead of two, and the training process arranges the words so that "near" means "used in similar contexts."

Word2Vec, GloVe, and fastText

Word2Vec made this practical. In a paper first posted in January 2013, Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean proposed two new architectures for learning word vectors and reported that it takes "less than a day to learn high quality word vectors from a 1.6 billion words data set." The two architectures are known as CBOW, which predicts a word from its surrounding words, and skip-gram, which predicts the surrounding words from one word. Neither prediction is the goal; the vectors learned along the way are.

The vectors turned out to carry structure. In a 2013 NAACL paper, Mikolov, Wen-tau Yih, and Geoffrey Zweig showed that vector offsets captured relationships: the vector for "king" minus "man" plus "woman" lands near "queen." The authors reported that their vectors answered syntactic analogy questions correctly almost 40% of the time. The famous example works because a consistent direction in the space corresponds to a relation such as gender, not because anyone programmed it.

Two variations followed. GloVe, from Jeffrey Pennington, Richard Socher, and Christopher Manning in 2014, trains on global word-word co-occurrence statistics gathered over a whole corpus. Its pretrained vectors, such as those trained on 840 billion tokens of Common Crawl web text, became a standard download. fastText, from Piotr Bojanowski and colleagues in 2016, treats each word as a bag of character n-grams, which lets it produce a vector for a word it never saw in training. For the modern descendants of these ideas, see our guide to embeddings and vector search.

The limit: one word, one vector

Static embeddings gave "bank" a single vector that blended the riverbank and the savings bank. That meant representing a word's meaning independent of its sentence, which is precisely what meaning is not. The fix had to make vectors depend on context.

Sequence models and attention (2014 to 2017)

Translation and summarization turn one sequence into another, so the field built models for that job. The standard design was an encoder-decoder, also called sequence-to-sequence. The encoder reads the input and compresses it into a vector. The decoder generates the output one word at a time from that vector.

The compression step was the flaw. Forcing a long sentence through one fixed-size vector is like summarizing a whole book onto a single sticky note and then asking someone to translate from the note. In a paper first posted in September 2014 and presented at ICLR 2015, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio argued that the fixed-length vector is a bottleneck. Their remedy let the model "automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word." In the book analogy, the translator gets to glance back at the pages while writing each word.

The production payoff arrived quickly. In September 2016, Google researchers reported that their neural system reduced translation errors by an average of 60% compared with the company's phrase-based production system, in a human side-by-side evaluation on isolated simple sentences. The qualifier matters: simple sentences are easier than real documents, so the figure describes a favorable test.

The Transformer

Recurrent networks, which read one word at a time, were slow to train because each step depended on the previous one. In June 2017, a team at Google proposed dropping recurrence and relying on attention alone. The paper Attention Is All You Need reported 28.4 BLEU on WMT 2014 English-to-German, more than 2 BLEU above the previous best including ensembles, and 41.8 BLEU on English-to-French, after training for 3.5 days on eight GPUs. Because attention processes all the words of a sentence in parallel, the design could be trained on far more data, and nearly every major language model since has used it. The details are in our explainers on how Transformers pay attention to words and how large language models work.

Read first, specialize later (2018 onward)

The next step combined two ideas: contextual vectors and pretraining on unlabeled text. Pretraining means first learning general language patterns from a huge pile of raw text, with no human labels, and then adapting the result to a specific task with a small labeled set. Our explainer on transfer learning covers why that saves so much effort.

In February 2018, a team led by Matthew Peters released deep contextualized word representations, later known as ELMo, in which each word's vector is "a learned function of the internal states of a deep bidirectional language model." Adding these vectors to existing systems improved the state of the art on six tasks. Context finally changed the vector: the two senses of "bank" got different representations.

BERT

In October 2018, Jacob Devlin and colleagues at Google posted BERT, short for Bidirectional Encoder Representations from Transformers. Its training used two self-supervised tasks:

• Masked language modeling. The model hides 15% of the word pieces in each input and predicts them from both sides of the context. Of the chosen positions, 80% are replaced with a special mask symbol, 10% with a random token, and 10% are left unchanged.
• Next sentence prediction. Given two sentences, the model predicts whether the second really followed the first in the source text. In training, that was true 50% of the time.

The pretraining corpus was BooksCorpus (800 million words) plus English Wikipedia (2,500 million words). BERT-base had 110 million parameters and BERT-large had 340 million. Fine-tuning for a specific task was cheap: the paper says all its results can be replicated in at most one hour on a single Cloud TPU, starting from the same pretrained model.

The gains were large. On the GLUE benchmark, a collection of language-understanding tasks, BERT-large scored 80.5 on the official leaderboard, 7.7 points above the previous best. It reached 86.7% on the MultiNLI inference task and an F1 of 93.2 on SQuAD v1.1 question answering.

System (GLUE test, average over tasks)	MNLI-m accuracy	Average
Previous state of the art	80.6	74.0
BiLSTM + ELMo + attention	76.4	71.0
OpenAI GPT	82.1	75.1
BERT-base	84.6	79.6
BERT-large	86.7	82.1

Source: Table 1 in the BERT paper. The average excludes one problematic GLUE task, so it differs slightly from the official GLUE score of 80.5.

A later paper, RoBERTa, concluded that "BERT was significantly undertrained" and that a better-tuned BERT could match models published after it. The lesson, that scale and training choices matter as much as architecture, shaped what came next. Our piece on neural scaling laws follows that thread.

Benchmarks saturate

GLUE was posted in April 2018. By May 2019, researchers had to build a successor. The SuperGLUE paper explained that performance on GLUE had "surpassed the level of non-expert humans," leaving little room for further progress. This pattern, in which a benchmark is solved soon after it is built, repeats across NLP. It is one reason the numbers in a table like the one above should be read as snapshots, a point our article on how AI benchmarks measure intelligence develops.

Scale and few-shot learning

In May 2020, OpenAI researchers described GPT-3, "an autoregressive language model with 175 billion parameters," ten times larger than any previous non-sparse language model. The paper's claim was that scale improves "few-shot" performance: given a handful of examples in the prompt, with no weight updates, the model could attempt new tasks. Our overview of large language models covers what followed.

Reasoning, sentiment, and where models still break

Fluent text is not the same as reliable text. Three areas show where the limits sit.

Reasoning

A 2022 paper by Jason Wei and colleagues, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, found that prompting a 540-billion-parameter model with eight worked examples that include intermediate steps achieved state-of-the-art accuracy on the GSM8K math word-problem benchmark. Writing out steps helps because each step becomes input to the next. Newer systems are trained to produce such steps on their own; our article on AI reasoning and chain of thought covers the evidence and the cases where the written steps do not reflect what the model actually did.

Sentiment analysis

Sentiment analysis labels the emotional tone of text as positive, negative, or neutral. In a 2002 paper, Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan used movie reviews to compare methods and found that standard machine learning methods definitively outperformed human-produced baselines, meaning word lists chosen by people. They also found that Naive Bayes, maximum entropy, and support vector machines all did worse on sentiment than on topic classification. Sentiment is harder than topic because the signal sits in how words combine: "not bad" and "terrible, but I loved it" defeat a simple tally.

Problems that remain include domain shift (a model trained on movie reviews may stumble on restaurant reviews), implied sentiment ("the food was slow"), and sarcasm. Financial applications raise the stakes, as our article on sentiment analysis for stocks explains.

Hallucination

A model trained to produce plausible next words will produce plausible text whether or not it is true. Our explainer on why AI hallucinates facts covers the causes. The practical rule is the one the ELIZA story taught in 1966: fluency is not evidence of comprehension or accuracy.

What people get wrong about NLP's history

Myth: each era erased the one before. Parts of the earlier toolkits survive inside modern systems. Byte pair encoding, the tokenizer behind most large models, is a statistical method that learns which character sequences are frequent. The dependency and constituency labels from linguistics still define what parsers are trained to produce. Manning's 2011 tagging paper was subtitled "Is It Time for Some Linguistics?" and argued that the largest remaining opportunity lay in improving the linguistic categories that taggers learn from.

Myth: understanding was settled by the Turing test. Passing a five-minute chat test shows that a system can sustain a convincing conversation. It does not settle whether the system models the world behind the words, which is why researchers keep building harder evaluations.

Myth: bigger always wins. Scale mattered enormously, but RoBERTa showed that a better training recipe on the same architecture could close a lot of ground, and the Transformer's original result came from an architecture change on a budget of 3.5 days and eight GPUs.

Do language models understand language?

The honest answer is that the question splits in two. Operationally, modern models translate, summarize, answer questions, and write code at a level that was out of reach a decade ago, as the benchmark history above shows. Philosophically, whether predicting text amounts to understanding it is contested, and the ELIZA effect is a standing warning that humans are generous interpreters.

A fair summary is that behavior and mechanism need separate evidence. Behavior can be tested with benchmarks, though benchmarks saturate and can be gamed. Mechanism, meaning what happens inside the network, is the subject of interpretability research, which is still young. Until that work matures, claims in either direction, that these systems "just autocomplete" or that they "truly understand," outrun the evidence.

The takeaway

NLP's history is a series of decisions about where knowledge should live. First it lived in rules written by experts, which gave precision and brittleness. Then it lived in counts over text, which gave robustness and a short memory. Then it lived in vectors and attention learned from unlabeled data, which gave context and scale at the cost of transparency.

Each shift traded something. The pattern worth carrying forward is that the next step usually came from removing a human-written assumption, such as a grammar rule, a hand-built feature, or a fixed-length summary, and letting the model learn that piece from data. The open question is which assumption comes out next, and how we will check what the model learned in its place.

Frequently asked questions

Q: What was the first machine translation demonstration? A: The Georgetown-IBM experiment, demonstrated in January 1954, is the usual starting point. It was small, with 250 words and six grammar rules, but it raised expectations that high-quality automatic translation was close.

Q: Why did statistical NLP replace rule-based NLP? A: Rules had to be written by hand and broke on cases nobody anticipated. Statistical methods estimate behavior from large collections of real text, so they handle messy input more gracefully, and they improve when given more data instead of more rules.

Q: What is a word embedding? A: A word embedding represents a word as a list of numbers so that words used in similar contexts get similar lists. Word2Vec, published in 2013, made them practical: its authors reported learning high-quality vectors from 1.6 billion words in under a day.

Q: What did BERT change? A: BERT showed that a model pretrained on unlabeled text by masking words, then fine-tuned briefly on a labeled task, could beat task-specific systems. It scored 80.5 on GLUE, 7.7 points above the previous best, and fine-tuning took about an hour on a single TPU.

Q: Do language models actually understand language? A: It depends on what "understand" means. They perform well on translation, summarization, and question answering, but they also state falsehoods fluently and fail on some simple puzzles. Whether predicting text equals understanding it is a contested question.

Q: Why do language models sometimes make obvious mistakes? A: They are trained to produce likely text, not verified text. When training data contains errors, or when a question differs from what the model has seen, fluent wrong answers can result. Our article on why AI hallucinates covers the causes in detail.

---

Read next:

• AI Bias and Discrimination: How Algorithms Fail Us
• Fine-Tuning LLMs Explained
• Context Window Explained

Sources

• Turing, A. (1950). Computing Machinery and Intelligence. Mind, 59(236), 433-460.
• Jones and Bergen (2025). Large Language Models Pass the Turing Test
• Hutchins (2004). The Georgetown-IBM experiment demonstrated in January 1954
• ALPAC, committee overview
• Brown et al. (1990). A Statistical Approach to Machine Translation
• Manning (2011). Part-of-Speech Tagging from 97% to 100%
• Sennrich et al. (2015). Neural Machine Translation of Rare Words with Subword Units
• Mikolov et al. (2013). Efficient Estimation of Word Representations in Vector Space
• Mikolov, Yih, and Zweig (2013). Linguistic Regularities in Continuous Space Word Representations
• Pennington et al. (2014). GloVe
• Bojanowski et al. (2016). Enriching Word Vectors with Subword Information
• Bahdanau et al. (2014). Neural Machine Translation by Jointly Learning to Align and Translate
• Wu et al. (2016). Google's Neural Machine Translation System
• Vaswani et al. (2017). Attention Is All You Need
• Peters et al. (2018). Deep contextualized word representations
• Devlin et al. (2018). BERT
• Liu et al. (2019). RoBERTa
• Wang et al. (2019). SuperGLUE
• Brown et al. (2020). Language Models are Few-Shot Learners
• Wei et al. (2022). Chain-of-Thought Prompting
• Pang, Lee, and Vaithyanathan (2002). Thumbs up? Sentiment Classification using Machine Learning Techniques