How Large Language Models Work: Transformers, Attention, and Why They Hallucinate

Meta's Llama 3 models were pre-trained on more than 15 trillion tokens of text, and the 70-billion-parameter version alone consumed about 6.4 million GPU-hours on Nvidia H100 chips, according to Meta's published model card. All of that computation serves a single, almost absurdly simple objective: guess the next piece of text. Yet a system trained only to guess the next word can translate, summarise, write code and explain physics — and it can also state a wrong answer with total confidence. Understanding both halves of that story requires understanding what large language models (LLMs) actually do, which turns out to be simpler and stranger than most people assume.

Key takeaways

• LLMs predict tokens: At their core, language models assign probabilities to what token (a word or word fragment) comes next, then repeat.
• Transformers replaced RNNs: The 2017 paper "Attention Is All You Need" introduced the architecture now used by virtually every major LLM.
• Attention is selective memory: Self-attention lets each token look at every earlier token in the context and decide which relationships matter most.
• Scale is predictable — mostly: Loss falls as a smooth power law with compute, parameters and data. Whether some abilities "emerge" suddenly is disputed.
• Preference tuning shapes behaviour: RLHF and newer methods like DPO turn a raw text predictor into a helpful assistant.
• Hallucination is structural: LLMs do not retrieve facts; they generate plausible continuations. That is different from a database lookup.

The Fundamental Operation: Next-Token Prediction

Strip away the complexity and an LLM does one thing: given a sequence of tokens, it outputs a probability distribution over what the next token should be. A token is a chunk of text from a fixed vocabulary — for English, common words are often a single token, while rarer words are split into pieces, so "unbelievable" might become "un", "believ" and "able". As a rough rule of thumb, an English token averages about three to four characters.

During pre-training, the model reads a stretch of real text and, at every position, tries to predict the token that actually follows. The difference between its predicted distribution and the true answer is measured with a loss function called cross-entropy. That error signal is propagated backward through the network (backpropagation) and every parameter is nudged slightly in the direction that would have made the right answer more likely. Repeat this across trillions of tokens of varied text and something remarkable happens: to keep lowering its prediction error, the model has to absorb grammar, facts, styles of argument, programming conventions and social norms — because all of those shape which word comes next in human writing.

This is called self-supervised learning. No human has to label anything; the text is its own answer key. That is the key economic fact behind LLMs: the training signal is essentially free, so the only limits are how much text you can collect and how much compute you can afford. If you want the general background on how models learn from data, see what machine learning is, explained simply.

A worked example of generation

Suppose the prompt is "The capital of France is". The model converts it to token IDs, runs them through the network, and produces a score (a logit) for every token in its vocabulary — often tens of thousands of candidates. A softmax function turns those scores into probabilities. " Paris" might get 0.92, " a" 0.02, " located" 0.01, and so on. The decoding algorithm picks one token, appends it to the input, and runs the whole network again to predict the token after that. A 500-word answer is therefore hundreds of separate forward passes, each conditioned on everything generated so far. This loop is why the model is called autoregressive, and why responses stream out word by word.

From Words to Numbers: Tokenization and Embeddings

Neural networks operate on numbers, not letters, so text goes through two conversions before the transformer sees it.

Tokenization. Most LLMs use a subword algorithm such as Byte Pair Encoding (BPE) or SentencePiece. BPE starts with individual bytes or characters and repeatedly merges the most frequent adjacent pair into a new vocabulary entry until it reaches a target size. Common vocabularies range from roughly 32,000 to over 100,000 entries; Meta's Llama 3, for example, moved to a 128,000-token vocabulary. Tokenization explains several quirks people notice: models are often bad at counting letters in a word (they never see the letters, only the chunks), and languages under-represented in the tokenizer's training data need more tokens per sentence, which makes them slower and more expensive to process.

Embeddings. Each token ID is then mapped to a vector — a list of a few thousand numbers — by looking it up in a learned embedding table. Tokens used in similar contexts end up with similar vectors, so the geometry of this space encodes meaning. Because a transformer processes all positions in parallel, it also needs to know word order, so a positional signal is added or applied; many modern models use rotary position embeddings (RoPE), which rotate the query and key vectors by an angle that depends on position. The same vector idea powers semantic search, covered in our explainer on embeddings and vector search.

The Transformer Architecture

Before 2017, the leading language models were recurrent neural networks (RNNs) and their gated variant, LSTMs. They processed text sequentially, one token at a time, carrying a fixed-size hidden state forward. That made training slow — you cannot compute step 500 until step 499 is done — and it made long-range dependencies hard to learn, because information from early tokens had to survive hundreds of updates to that single state.

The transformer, introduced by Ashish Vaswani and seven co-authors at Google in the paper "Attention Is All You Need" (first posted June 12, 2017), discarded recurrence entirely. Every token can attend directly to every other token, and all positions are computed in parallel on GPUs. The original model was built for machine translation and set a new state of the art on the WMT 2014 English-to-French benchmark (41.8 BLEU) after training for 3.5 days on eight GPUs — a tiny budget by today's standards.

A transformer is a stack of identical layers (blocks). Each block contains two main components, each wrapped with a residual connection and layer normalisation:

Multi-head self-attention. For each token, the model computes three vectors — Query (Q), Key (K) and Value (V) — by multiplying the token's current representation by three learned weight matrices. The attention weights between tokens are softmax(QKᵀ / √d_k), where d_k is the key dimension; the scaling by √d_k keeps the dot products from growing so large that the softmax saturates. Each token's output is the weighted sum of all Value vectors, so it "borrows" information from the tokens it attends to most.

Feed-forward network (MLP). After attention, each token's vector passes independently through a two-layer neural network that expands it to several times its width and projects it back. Interpretability research suggests a large share of a model's stored factual associations live in these MLP layers, while attention mainly moves information between positions.

The output of one block becomes the input of the next. Large models stack dozens of blocks; GPT-3, for instance, used 96 layers. A useful mental model is a residual stream: a running vector for each token that every layer reads from and writes a small update into. By the final layer, the vector at the last position holds everything the model needs to score the next token. For a gentler introduction to the layers and weights involved, see how neural networks work.

Encoder, decoder, or both

The original transformer had an encoder (which reads the input) and a decoder (which writes the output). Later families specialised:

Family	Example	Attention pattern	Typical use
Encoder-only	BERT (2018)	Bidirectional — every token sees the whole sentence	Classification, search ranking, embeddings
Encoder-decoder	T5 (2019)	Encoder bidirectional, decoder causal	Translation, summarisation
Decoder-only	GPT series, Llama	Causal — each token sees only earlier tokens	Open-ended generation, chat assistants

Decoder-only models add causal masking: each token is blocked from attending to future tokens. This enforces left-to-right generation and lets the same network be trained on every position of a document in one pass. Almost every chat assistant in use today is a decoder-only transformer.

Why Attention Works: An Intuition

Consider the sentence "The trophy didn't fit in the suitcase because it was too big." What does "it" refer to? A human resolves this instantly: the trophy. Change "big" to "small" and "it" becomes the suitcase. Resolving the pronoun requires world knowledge about sizes and containers, combined with the specific words in this sentence.

With self-attention, the representation of "it" can query every earlier token. If the learned Query for "it" matches the Keys of "trophy" strongly, the model pulls in information from "trophy" — and the attention pattern can depend on "big" versus "small" because those words have already shaped the representations in earlier layers. Nobody programs these rules; they are learned because they reduce prediction error across billions of examples.

"Multi-head" attention runs many of these lookups in parallel, each with its own Q, K and V matrices. GPT-3's largest model used 96 attention heads per layer. Different heads learn different kinds of relationship: some track the previous token, some link a pronoun to its antecedent, some match an opening bracket to its closing one. A well-studied example is the induction head — a two-head circuit that spots an earlier occurrence of the current token and copies whatever followed it, a mechanism thought to underpin much of in-context learning. Our dedicated piece on how transformers pay attention to words walks through attention maps visually.

Scale, Scaling Laws, and the "Emergence" Debate

GPT-2 (2019) had 1.5 billion parameters. GPT-3, described in the 2020 paper "Language Models are Few-Shot Learners", had 175 billion — which the authors noted was ten times more than any previous non-sparse language model. The headline result was few-shot in-context learning: show the model a handful of examples in the prompt and it would perform the task without any change to its weights.

Two lines of research tried to make the effect of scale predictable:

• Kaplan et al. (OpenAI, 2020) reported that test loss falls as a smooth power law in model size, dataset size and compute, over many orders of magnitude.
• Hoffmann et al. (DeepMind, 2022) — the "Chinchilla" paper — found that for a fixed compute budget, model size and training tokens should be scaled in equal proportion. Their 70-billion-parameter Chinchilla, trained on about four times more data than the 280-billion-parameter Gopher with the same compute, outperformed it. The conclusion: many earlier large models had been undertrained.

The industry took the lesson further. Meta's Llama 3 8B was trained on more than 15 trillion tokens — far beyond the Chinchilla-optimal amount for a model that size — because a smaller model that has seen more data is cheaper to run for millions of users, even if training cost more than strictly necessary.

Are emergent abilities real?

A 2022 line of work described "emergent abilities" — skills such as multi-digit arithmetic that seemed near zero in small models and then jumped sharply at larger scale. In 2023, Rylan Schaeffer, Brando Miranda and Sanmi Koyejo argued in "Are Emergent Abilities of Large Language Models a Mirage?" that many of these jumps come from the choice of metric. An all-or-nothing score like exact match can make steady underlying improvement look like a sudden leap; switch to a continuous metric and the curve often becomes smooth. The debate is not fully settled, but the safest statement today is that overall loss improves predictably with scale, while how that loss translates into specific skills is harder to forecast.

Training: From Raw Data to Language Model

Building a frontier model is an industrial process with several distinct stages.

Data curation. Pre-training corpora mix filtered web crawls (Common Crawl is the most common base), books, encyclopedias, scientific papers and source code. Raw web text is heavily filtered: duplicate pages are removed, low-quality and boilerplate text is discarded with classifiers, and personal data and toxic content are reduced. The mix matters enormously — developers widely report that adding code improves structured reasoning, and the proportion of each source is one of the most closely guarded details of commercial models.

Distributed training. No single GPU can hold a model with tens of billions of parameters plus its optimiser state, so training is split across thousands of accelerators using three complementary techniques: data parallelism (different batches on different GPUs), pipeline parallelism (different layers on different GPUs) and tensor parallelism (single matrix multiplications split across GPUs). According to Meta's Llama 3 model card, the 8B model took 1.3 million and the 70B model 6.4 million H100-80GB GPU-hours, with estimated emissions of about 390 and 1,900 tonnes of CO₂-equivalent respectively (which Meta says it offset).

Optimisation. Training typically uses the AdamW optimiser with a learning-rate warm-up followed by gradual decay, gradient clipping to prevent rare huge updates from destabilising the run, and mixed-precision arithmetic (bfloat16) to halve memory use. Even so, long runs suffer loss spikes and hardware failures, so checkpoints are saved frequently and runs are restarted from the last good state.

Model	Year	Parameters	Training data	Source
Original Transformer (big)	2017	— (translation model)	WMT 2014 pairs; 3.5 days on 8 GPUs	Vaswani et al.
GPT-3	2020	175 billion	Not stated in abstract	Brown et al.
Gopher	2021	280 billion	Same compute as Chinchilla	Hoffmann et al.
Chinchilla	2022	70 billion	~4× Gopher's data	Hoffmann et al.
Llama 3 8B / 70B	2024	8 / 70 billion	15T+ tokens; 1.3M / 6.4M H100 GPU-hours	Meta model card

From Text Predictor to Assistant: SFT, RLHF and DPO

A freshly pre-trained model is a powerful but awkward tool. Ask it a question and it may continue with three more questions, because that is what often follows a question on a web forum. It has no particular inclination to be helpful, honest or safe; it simply continues text.

OpenAI's 2022 InstructGPT paper, "Training language models to follow instructions with human feedback", laid out the recipe most assistants still follow in some form:

1. Supervised fine-tuning (SFT). Human labelers write demonstration answers to prompts, and the model is fine-tuned on these pairs. This teaches the format of a helpful response.
2. Reward model training. Labelers rank several model outputs for the same prompt. These rankings train a separate reward model that scores any response.
3. Reinforcement learning. The SFT model is optimised with Proximal Policy Optimization (PPO) to maximise the reward model's score, with a KL-divergence penalty that keeps it from drifting too far from the SFT model and exploiting quirks in the reward model.

The striking result: human evaluators preferred outputs from a 1.3-billion-parameter InstructGPT over the 175-billion-parameter GPT-3, despite the hundredfold size gap. Alignment tuning does not add much knowledge; it changes how existing knowledge is expressed.

PPO-based RLHF is finicky to run, so newer methods simplify it. Direct Preference Optimization (DPO), introduced by Rafailov and colleagues in a 2023 paper, showed that the RLHF objective can be solved with a simple classification-style loss directly on preference pairs, with no separate reward model and no sampling during training, matching or beating PPO in their experiments. Anthropic's Constitutional AI replaces many human comparisons with AI-generated feedback guided by a written set of principles (often called RLAIF). The trade-offs between these approaches are compared in RLHF vs RLAIF vs DPO.

Fine-tuning more cheaply. Not every adaptation needs full retraining. LoRA (Low-Rank Adaptation, Hu et al., 2021) freezes the original weights and learns small low-rank update matrices instead; on GPT-3 175B the authors reported about 10,000 times fewer trainable parameters and roughly a third of the GPU memory compared with full fine-tuning using Adam. That is why hobbyists can customise open models on a single consumer GPU.

Decoding: How the Model Chooses Each Word

The model outputs probabilities; a separate decoding strategy turns them into text. The choice has a large effect on output quality.

• Greedy decoding always takes the most probable token. It is deterministic but tends toward repetitive, bland text.
• Temperature divides the logits by a constant before the softmax. Values below 1 sharpen the distribution (more predictable); values above 1 flatten it (more varied, more error-prone). Temperature 0 is treated as greedy.
• Top-k sampling samples only from the k most probable tokens.
• Top-p (nucleus) sampling samples from the smallest set of tokens whose probabilities sum to p, such as 0.9 — the candidate pool shrinks when the model is confident and widens when it is not.

None of these settings makes the model more knowledgeable. A low temperature only makes it repeat its single best guess more consistently; if that guess is wrong, it will be consistently wrong. How phrasing the prompt itself steers these probabilities is covered in prompt engineering explained.

Why LLMs Hallucinate

This is the most important limitation to understand. An LLM does not store facts as rows in a table. It compresses statistical regularities of its training text into billions of weights. When asked "Who invented the telephone?", it does not look up a record; it generates the most probable continuation given those weights. "Alexander Graham Bell" is strongly associated with "telephone" in the training data, so the answer is usually right. The same process applied to an obscure scientist, a recent court case, or a citation with a specific page number produces text that has the shape of a correct answer, with no internal check on whether it is true.

Several factors make this worse:

Sparse training signal. A fact that appeared only a few times in the training data leaves a weak trace, and the model may blend it with more common, similar-looking patterns — mixing up two researchers with similar names, for example.

Training cutoff. The model knows nothing after its data was collected unless that information is supplied in the prompt. Asked about later events, it may extrapolate confidently.

Training incentives. Pre-training rewards fluent continuation, not abstention. Unless later tuning explicitly rewards saying "I don't know", a guess usually scores better than silence.

Weak calibration. A token's probability is not the same as the probability that the overall claim is true. A model can be highly confident in each word of a fabricated reference.

Retrieval-augmented generation (RAG), introduced by Patrick Lewis and colleagues in a 2020 NeurIPS paper, reduces the problem by retrieving relevant documents from an external index and placing them in the prompt, so the model can ground its answer in a source. It helps, but does not eliminate the problem: retrieval can fetch the wrong document, and the model can still misread or ignore what it was given. For a deeper look at failure modes, see why AI hallucinates facts, and for the retrieval architecture itself, what RAG is.

Common misconceptions

Misconception	What actually happens
"The model searches the internet when it answers."	A plain LLM uses only its weights and the prompt. Web access is a separate tool bolted on by the application.
"It remembers our past conversations."	Within a session, earlier messages are re-sent as part of the context. Across sessions, memory exists only if the product stores and re-inserts it.
"It learns from every chat in real time."	The weights are fixed after training. Any later improvement comes from a new training run.
"Low temperature makes it accurate."	It makes it consistent. Accuracy depends on what the model learned and what is in the context.
"It is copying a stored document."	Mostly it generates new sequences, though models can memorise and regurgitate text that appeared many times in training.

The Context Window: Working Memory

A transformer can only attend to a fixed maximum number of tokens at once — its context window. GPT-3 used a 2,048-token window; Llama 3 used 8,192 tokens at release; and commercial models have since advertised windows of hundreds of thousands to a million tokens. In standard attention, every token is compared with every other, so compute and memory for the attention step grow with the square of the context length. Long-context models rely on engineering such as memory-efficient attention kernels and caching of previously computed keys and values (the KV cache) to make this tractable.

Inside the window, the model has something like working memory: it can reference any earlier token. Outside it, there is nothing, unless the application adds external memory such as a vector database. A bigger window is also not the same as perfect recall. In "Lost in the Middle" (Liu et al., 2023), researchers found that models used information best when it sat near the start or end of a long input, and performance degraded significantly when the key passage was buried in the middle. Practical advice follows directly: put the most important instructions and evidence at the beginning or end. Our context window explainer covers the trade-offs in more depth.

Can LLMs Reason?

LLMs pass many reasoning benchmarks, and prompting a model to write out intermediate steps ("chain of thought") substantially improves performance on multi-step maths and logic problems. Newer "reasoning models" are trained, often with reinforcement learning on verifiable problems, to generate long internal chains of reasoning before answering. At the same time, researchers keep finding brittle failures: changing irrelevant names or numbers in a word problem can lower accuracy, which suggests at least part of the performance relies on pattern-matching to familiar problem templates. A fair summary is that LLMs perform real, useful multi-step computation, but not the guaranteed, rule-following inference of a formal logic system. We look at this question in detail in how LLMs actually reason.

Limitations and Open Questions

Several fundamental questions remain open as of 2026:

• Interpretability. Researchers can identify some circuits and features inside models, but cannot yet fully explain why a large model produces a given answer.
• Data limits. Training sets are measured in trillions of tokens, and the stock of high-quality public human text is finite. How far synthetic data can substitute is an active research question.
• Factual grounding. No current method guarantees a model will not fabricate; retrieval, citations and verification tools reduce the rate rather than eliminate it.
• Cost and energy. Training and serving large models consume substantial electricity, as the CO₂ figures in Meta's model card illustrate.
• Evaluation. Benchmarks leak into training data and saturate quickly, which makes it hard to measure genuine progress.

FAQ

Q: What is the difference between a language model and a chatbot? A: The language model is the trained network that predicts tokens. A chatbot is an application built on top of it, adding a system prompt, conversation formatting, safety filters, and often tools like web search or file reading. The same underlying model can power many different products.

Q: How many parameters does GPT-4 have? A: OpenAI has not officially disclosed GPT-4's size or architecture. Figures circulating online are unconfirmed estimates and should not be treated as fact. Published parameter counts are available for open models such as Llama 3 (8 and 70 billion) and for earlier systems such as GPT-3 (175 billion).

Q: What does "temperature" do? A: Temperature scales the raw scores (logits) before they are converted into probabilities. Higher values flatten the distribution, producing more varied and more error-prone text; lower values sharpen it so the top-scoring token wins almost every time. Temperature 0 is treated as greedy decoding. It changes variety, not knowledge.

Q: Why are LLMs so computationally expensive to run? A: Each new token requires a full forward pass through every layer. A useful rule of thumb is about two floating-point operations per parameter per token, so a 175-billion-parameter model needs roughly 350 billion operations for every token it produces. Serving millions of users therefore requires large fleets of specialised accelerators.

Q: What is the difference between pre-training and fine-tuning? A: Pre-training teaches general language patterns from trillions of tokens and costs millions of GPU-hours. Fine-tuning adapts the pre-trained model to a task or behaviour with a much smaller dataset. Methods like LoRA make fine-tuning cheaper still by training small add-on matrices while the original weights stay frozen.

Q: Will LLMs keep getting better with more scale? A: Scaling laws show loss falling predictably as compute, parameters and data grow, but each improvement costs more, and high-quality data is finite. Much recent progress has come from better data, longer training of smaller models, preference tuning and reasoning-focused training rather than raw parameter count alone.

Q: Does a bigger context window mean the model remembers everything I paste in? A: Not reliably. Studies such as "Lost in the Middle" found that models use information near the start or end of a long input better than information in the middle. Putting key instructions and evidence at the beginning or end of a prompt tends to work better.

Sources

• Vaswani et al., "Attention Is All You Need" (2017): https://arxiv.org/abs/1706.03762
• Brown et al., "Language Models are Few-Shot Learners" (2020): https://arxiv.org/abs/2005.14165
• Hoffmann et al., "Training Compute-Optimal Large Language Models" (2022): https://arxiv.org/abs/2203.15556
• Ouyang et al., "Training language models to follow instructions with human feedback" (2022): https://arxiv.org/abs/2203.02155
• Rafailov et al., "Direct Preference Optimization" (2023): https://arxiv.org/abs/2305.18290
• Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (2021): https://arxiv.org/abs/2106.09685
• Schaeffer, Miranda & Koyejo, "Are Emergent Abilities of Large Language Models a Mirage?" (2023): https://arxiv.org/abs/2304.15004
• Liu et al., "Lost in the Middle" (2023): https://arxiv.org/abs/2307.03172
• Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (2020): https://arxiv.org/abs/2005.11401
• Meta, Llama 3 model card (2024): https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

---

Read next:

• How Does ChatGPT Actually Work? A Plain-Language Guide to LLMs
• Chatbots Explained — From ELIZA to ChatGPT
• Prompt Caching Explained — Cut LLM Costs