How Transformers Pay Attention To Words

In the sentence "The trophy didn't fit in the suitcase because it was too big," what does "it" refer to, the trophy or the suitcase? Humans resolve this instantly using context accumulated across the whole sentence, not just the words immediately next to "it." Getting a neural network to do the same thing, reliably, across arbitrarily long and structurally messy sentences, is exactly the problem that self-attention was built to solve. Eight researchers at Google published the solution in June 2017, in a paper with an unusually blunt title: "Attention Is All You Need." That paper introduced the transformer architecture, and nearly every large language model built since, GPT, BERT, Claude, Gemini, and the rest, is a descendant of the mechanism it described (Vaswani et al., 2017, arXiv:1706.03762).

Before transformers: the sequential bottleneck

For most of the 2010s, the dominant architectures for processing language were recurrent neural networks (RNNs) and a more capable variant called long short-term memory networks (LSTMs). Both worked by reading a sentence strictly in order, one token at a time, updating a single running summary, called a hidden state, after each word. By the time the network reached the end of a long sentence, that hidden state was supposed to contain everything relevant from every word that came before it.

That design has an obvious weakness. A hidden state is a fixed-size vector, a list of numbers of constant length, and cramming an ever-growing amount of context into a container that never grows means older information gets overwritten or diluted as the sequence continues. Researchers called the resulting failure mode the vanishing gradient problem: during training, the signal telling the network how an early word should have influenced a late prediction shrinks as it propagates backward through dozens or hundreds of sequential steps, until it's too weak to teach the model anything useful about long-range relationships.

LSTMs, introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997, mitigated this with gating mechanisms that let the network selectively preserve or discard information, and they were a genuine improvement. But the core constraint remained: processing was sequential. Word 500 in a sentence couldn't be computed until words 1 through 499 had already been processed in order, one step at a time. That sequential dependency did two kinds of damage. It made long-range dependencies hard to learn, since information from early words still had to survive a long relay chain to matter later. And it made training slow, because sequential steps can't be parallelized: a GPU built to perform thousands of calculations simultaneously was forced to wait for step 1 to finish before starting step 2, leaving most of its computational capacity idle.

Researchers had already discovered that attention, letting a model look back at earlier words when generating a later one, helped with translation quality, and by the mid-2010s attention layers were commonly bolted onto RNN-based encoder-decoder models to give them a way to reference distant context directly. The Google team's 2017 insight was more radical: what if you removed the recurrence entirely and built a model out of attention alone?

Self-attention: every word looks at every other word at once

Self-attention, the mechanism at the heart of the transformer, lets every token in a sequence directly compare itself against every other token in that same sequence, all at once, rather than waiting for information to arrive through a chain of sequential steps. (A token is the basic unit a language model processes, usually a word or a word-fragment, since long or rare words often get split into smaller pieces during preprocessing.) For each pair of tokens, the mechanism computes an attention score, a single number representing how relevant one token is to interpreting another.

In the trophy-and-suitcase sentence, the token "it" can attend directly to both "trophy" and "suitcase" in a single computational step, with no intervening words to pass through. During training on large amounts of text, the model learns statistical patterns, such as the fact that "didn't fit because it was too big" more often describes the contained object than the container, and it uses that learned pattern to weight "trophy" more heavily than "suitcase" when building its internal representation of "it." Crucially, this isn't a hand-coded grammar rule. It emerges from the attention weights the model learns during training, the same way the rest of its knowledge does.

This "all tokens at once" property is what makes self-attention parallelizable in a way sequential recurrence never could be. Every attention score in a sentence can, in principle, be computed simultaneously on separate cores of a GPU, because none of them depend on the others being finished first. That single architectural choice, trading a sequential chain for a set of simultaneous pairwise comparisons, is the change that made the rest of the transformer's advantages possible.

"Attention Is All You Need" (Vaswani et al., Google Brain and Google Research, 2017) has become one of the most cited papers in the history of computer science, and its transformer architecture underlies nearly every major large language model released since, including GPT, BERT, Claude, and Gemini.

The mechanics: queries, keys, and values

The paper formalizes self-attention through three learned vector representations that every token gets, called a query, a key, and a value. Each is produced by multiplying the token's embedding, its numerical representation, by a separate weight matrix that the model learns during training. Conceptually, the query represents what a token is "looking for" in the rest of the sentence, the key represents what a token "offers" to other tokens that might be looking for it, and the value represents the actual content that gets passed along once relevance has been established.

The attention score between two tokens is computed by taking the dot product of one token's query with another token's key, a mathematical operation that produces a single number measuring how well the two vectors align. In "The trophy didn't fit in the suitcase because it was too big," the query vector for "it" is compared against the key vectors for every other token in the sentence, including "trophy" and "suitcase." Those raw scores are then scaled and passed through a softmax function, a mathematical operation that converts a list of numbers into probabilities that sum to one, turning the raw scores into a clean set of attention weights.

The final step blends the value vectors of every token in the sentence, weighted by those attention scores, into a new, updated representation for "it," one that now encodes relevant information pulled from wherever it's most useful across the sentence, not merely from adjacent words. The paper calls this process "scaled dot-product attention," and every token in the sequence goes through it simultaneously, each producing its own updated, context-enriched representation in a single parallel pass.

Why three separate vectors instead of one

Splitting each token into a query, a key, and a value, rather than comparing raw embeddings directly, gives the model three independent sets of learnable parameters instead of one, so it can separately learn what to search for, what to advertise, and what to actually pass along once a match is found. This separation gives the model more flexibility to represent different kinds of relationships between words than a single shared vector could support.

Multi-head attention: several perspectives on the same sentence

A single attention computation captures one kind of relationship well, but sentences carry many overlapping kinds of relationships simultaneously, grammatical structure, coreference (which pronoun refers to which noun), semantic similarity, and more. The transformer paper's answer to this was multi-head attention: instead of running self-attention once per layer, the model runs it several times in parallel, each with its own independently learned query, key, and value weight matrices, called "heads."

In the base configuration described in the original paper, the model uses eight parallel attention heads per layer, each with a smaller dimensionality so the total computation stays comparable to a single larger attention operation (Vaswani et al., 2017). Each head is free to specialize during training. One head might learn to track subject-verb agreement, another might specialize in resolving pronouns like "it" back to their referents, and another might focus on adjacent-word relationships similar to what older n-gram models captured. The outputs of all heads are concatenated, stacked together into one longer vector, and then passed through one more learned transformation to produce the layer's final output.

This "several perspectives at once" design is the transformer's second mandatory analogy: think of multi-head attention like a group of editors reading the same manuscript, one checking grammar, one checking factual consistency, one tracking character references across chapters, each producing separate notes that get merged into one final markup. No single editor catches everything, but the combination does. Multi-head attention gives the transformer that same division of labor, computed in parallel rather than handed off sequentially, and it's a big part of why transformers capture subtler patterns in text than a single attention pass could.

Positional encoding: giving a parallel model a sense of order

Removing recurrence solved the speed and long-range dependency problems, but it created a new one: without any sequential processing, the model has no inherent way to know that "dog bites man" and "man bites dog" describe different events, since self-attention treats a sentence as an unordered set of tokens being compared pairwise. Word order carries meaning in language, and a purely parallel architecture needed some other mechanism to encode it.

The paper's solution is positional encoding: a fixed mathematical pattern, generated using sine and cosine functions at different frequencies, that gets added directly to each token's embedding before it enters the first attention layer. Concretely, for a token at position "pos" and embedding dimension "i," the encoding uses sin(pos / 10000^(2i/d)) for even dimensions and cos(pos / 10000^(2i/d)) for odd ones, where d is the model's embedding size (Vaswani et al., 2017). The wavelengths of these sine and cosine waves form a geometric progression, so nearby positions produce similar encodings and distant positions produce increasingly different ones, giving the model a consistent, learnable signal about relative distance between any two tokens.

The authors chose this specific sinusoidal design, rather than simply assigning each position a unique learned number, because it lets the model generalize to sentence lengths it didn't see during training, and because the fixed mathematical relationship between positions makes it easier for attention layers to learn to attend by relative offset, such as "the word two positions to the left," a pattern useful across nearly any sentence structure. Later transformer variants have experimented with alternative schemes, including learned positional embeddings and rotary position embeddings, but the core requirement the original sinusoidal encoding solved, giving an otherwise order-blind, fully parallel architecture some notion of sequence, remains a fixed requirement of every transformer design since.

How positional encoding differs from the attention mechanism itself

Positional encoding is added once, to the input embeddings, before any attention computation happens; it isn't part of the query-key-value comparison itself. It functions less like a rule the model follows and more like a stamped timestamp on every token, information the attention layers can then choose to use, or partly ignore, as they learn which positional relationships actually matter for a given task.

How transformer attention compares to what came before

Self-attention wasn't competing against only recurrent networks. Convolutional neural networks (CNNs), architectures originally built for image processing, were also adapted for sequence modeling in the mid-2010s, using sliding filters over a sentence to capture local word patterns. Each of the three approaches makes a different trade-off between capturing long-range relationships and how efficiently the model can be trained.

Architecture	How it handles long-range dependencies	Parallelization during training	Representative use
RNN / LSTM	Sequential hidden state passed word to word; distant information decays over many steps	None — each step depends on the previous one finishing first	Pre-2017 machine translation, speech recognition
CNN-based (e.g., stacked convolutions)	Local filters see nearby words directly; distant relationships require stacking many layers to widen the effective range	High — filters at each position compute independently	Early neural sequence models, some fast translation systems
Transformer (self-attention)	Any token can directly attend to any other token in one step, regardless of distance	High — all pairwise attention scores computed simultaneously	GPT, BERT, Claude, Gemini, and nearly all modern large language models

Self-attention is the only one of the three architectures that gives every word a direct, one-step connection to every other word while still allowing the whole computation to run in parallel, which is the combination that made large-scale training practical.

Why this architecture triggered the scale-up era

The transformer's parallel structure matched modern GPU hardware in a way sequential recurrence never could. GPUs are built to perform enormous numbers of simultaneous matrix multiplications, and self-attention's pairwise token comparisons are, computationally, exactly that: a large matrix multiplication rather than a long chain of dependent steps. In the original paper, the authors trained their base model in roughly 12 hours and their larger model in about 3.5 days, on eight GPUs, reaching a then-state-of-the-art BLEU score of 28.4 on the WMT 2014 English-to-German translation benchmark and 41.0 on English-to-French, outperforming prior published single models at a fraction of the training cost researchers had come to expect for that level of quality (Vaswani et al., 2017).

That efficiency is what unlocked the scale-up that followed. Because transformers train faster per unit of compute than recurrent architectures at comparable quality, researchers could feed them dramatically more data and add more parameters, the model's internal learned numbers, without training runs becoming impractically slow. Google introduced BERT in 2018, a transformer encoder trained to understand text bidirectionally for tasks like search and classification. OpenAI's GPT series, built on the transformer's decoder half, followed the same scaling logic: GPT-2 in 2019, GPT-3 in 2020, and successive frontier models from OpenAI, Google, Anthropic, and others since, each substantially larger than the last. None of that trajectory would have been computationally realistic on a purely sequential architecture, since training time would have grown with sequence length in a way that made feeding models hundreds of billions of words of text impractical on any reasonable timeline.

It's worth being precise about what changed and what didn't: self-attention didn't invent the idea of scaling up neural networks, and larger recurrent models had been trained before 2017. What the transformer changed was the economics of scaling, making a given increase in model size and data volume dramatically cheaper to train in wall-clock time, which is what made the jump from research-lab-scale models to today's frontier models a realistic engineering project rather than a computationally prohibitive one.

The tradeoff self-attention introduces: quadratic cost

Self-attention's parallelism comes with a real computational cost that grows faster than the sequence itself. Because every token computes an attention score against every other token, the total number of pairwise comparisons scales quadratically with sequence length: doubling a document's length roughly quadruples the attention computation required, not merely doubles it. Processing a 100,000-token document therefore takes dramatically more compute per token than processing a 1,000-token one, and this quadratic relationship is the core technical reason context windows, the amount of text a model can consider at once, were historically capped far below what users wanted.

This bottleneck has driven a substantial line of follow-up research into more efficient attention variants. Sparse attention patterns restrict which token pairs get compared at all, skipping combinations judged unlikely to matter rather than computing every pair exhaustively. FlashAttention, introduced by researchers at Stanford in 2022, takes a different approach: it restructures how the underlying computation moves data through a GPU's memory hierarchy to avoid unnecessary reads and writes, producing mathematically identical attention results while running substantially faster and using less memory in practice (Dao et al., 2022, arXiv:2205.14135).

These optimizations, combined with other architectural refinements, are a major reason context windows expanded from roughly a few thousand tokens in early GPT-era models to hundreds of thousands or over a million tokens in current frontier models, even though the fundamental quadratic relationship between sequence length and raw attention cost hasn't been eliminated, only made cheaper to compute and manage in practice. Understanding why large models consume so much computing power in the first place starts here: attention's quadratic scaling is one of the largest single contributors to the compute budget of training and running a modern transformer.

What people get wrong about self-attention

A common misconception is that self-attention means the model "reads the whole sentence and understands it" the way a person does. What it actually does is compute a set of learned numerical weightings between token representations, weightings shaped entirely by patterns in training data, with no built-in comprehension, intent, or awareness of meaning the way human reading involves. The model has no separate concept of "the trophy is an object" and "the suitcase is a container" that it consults; it has only the statistical regularities its query, key, and value weights encode after training on enormous volumes of text.

A second frequent confusion conflates attention with the entire transformer architecture. Self-attention is one component, a critical one, but transformer blocks also include feed-forward layers, which apply the same learned transformation independently to each token's representation after attention has mixed in context, along with normalization steps and residual connections that keep training stable across many stacked layers. A full transformer model, like the ones behind GPT, Claude, or Gemini, typically stacks dozens of these blocks, each containing multi-head self-attention followed by a feed-forward layer, and it's the depth and scale of that stack, not attention alone, that produces the capabilities people associate with modern language models. Attention decides what to pay relevance to; the rest of the block decides what to do with that information.

The takeaway

Self-attention solves a concrete engineering problem that sequential models struggled with: letting any word in a sentence directly reference any other word, regardless of distance, without that information decaying through a long chain of intermediate steps, and doing so in a way that's fully parallelizable on modern hardware. That direct, parallel comparison, formalized as queries, keys, and values, multiplied across several attention heads, and grounded in position through sinusoidal encoding in the 2017 transformer paper, is the architectural foundation underneath essentially every major language model in use today. The tradeoff it introduced, attention cost that grows quadratically with sequence length, remains an active area of research nearly a decade later, which is itself a sign of how central the mechanism has become.

---

Read next:

• How AI models learn from feedback
• How AI image generators understand prompts
• Why AI needs so much computing power

FAQ

Q: What is self-attention in simple terms?

Self-attention lets every word in a sentence directly compare itself to every other word at once, computing a relevance score that determines how much each other word should influence its meaning, instead of processing the sentence strictly left to right the way older recurrent models did.

Q: What problem did transformers solve that earlier models couldn't?

Earlier recurrent models (RNNs and LSTMs) processed text sequentially and struggled to preserve information from early words by the time they reached later words in long sequences, a limitation researchers called the vanishing gradient problem. Self-attention connects distant words directly in a single step, avoiding that information decay, while also enabling parallel computation that sequential processing couldn't.

Q: What are queries, keys, and values in a transformer?

They are three learned vector representations of each token, produced by separate weight matrices. A query represents what a token is seeking, a key represents what it offers, and comparing queries against keys produces the attention scores used to blend value vectors into an updated, context-aware representation.

Q: What is multi-head attention and why does it matter?

Multi-head attention runs several independent self-attention computations in parallel, each with its own learned weights, so the model can track different kinds of relationships, such as grammar, coreference, or semantic similarity, simultaneously rather than relying on a single attention pass to capture everything at once.

Q: Why do transformers need positional encoding if they use attention?

Self-attention compares tokens pairwise without any inherent sense of order, so a transformer would otherwise treat a sentence as an unordered set of words. Positional encoding, a fixed sine-and-cosine pattern added to each token's embedding, gives the model information about word order and relative distance between tokens.

Q: Why did the transformer architecture become the standard for large language models?

Its parallel computation structure matches modern GPU hardware far better than the sequential structure of older recurrent models, allowing much faster training at much larger scale. That efficiency is what made it practical to train the massive models, GPT, BERT, Claude, and Gemini among them, that dominate language AI today.

Q: What is the main downside of self-attention?

Attention computation scales quadratically with sequence length, meaning doubling the input length roughly quadruples the compute required. This is why very long context windows are expensive to process and why researchers have developed techniques like sparse attention and FlashAttention to make long-context processing more efficient.