Mechanistic Interpretability: Reading an AI's Circuits

Mechanistic interpretability is the attempt to work out what a trained neural network is actually computing by examining its internal parts, the way an engineer might reverse-engineer a compiled program without the source code. Instead of asking a model why it said something and trusting the answer, researchers trace which internal features activated, which components passed information to which, and what happens to the output when a component is switched off. This article covers the vocabulary (features, circuits, superposition), the main results so far, and the limits that still keep the field short of a full explanation of any large model.

The black-box problem, stated precisely

A modern language model is a long chain of arithmetic. Text becomes numbers, the numbers pass through dozens of layers of matrix multiplication and nonlinear functions, and a probability distribution over the next token comes out the other end. Every one of those operations is visible. Nothing is hidden in the sense of being secret: you can print out every weight in a model and every intermediate value for any input.

The problem is that the printout means nothing to a human reader. A model like those described in how large language models work holds billions of numbers, and no single number carries an obvious meaning. Knowing that neuron 1,847 in layer 12 fired at 0.63 tells you roughly as much as knowing that one transistor in a CPU was at 0.9 volts while it rendered this web page.

Behavioral testing, the usual way to evaluate a model, has a different weakness. Giving a model thousands of questions and grading the answers tells you what it does on those questions. It cannot tell you what the model would do on the question nobody thought to ask, or whether a correct answer came from sound internal reasoning or from a shortcut that happens to work on the test set. A student who memorized last year's exam and a student who understands the subject look identical until the exam changes.

Mechanistic interpretability goes after the second thing: the algorithm inside. The word "mechanistic" signals the goal, which is a causal account of how inputs become outputs, in the style of a mechanism in biology (a signaling pathway, say) rather than a statistical correlation between inputs and outputs. If a claim about a model's internals is correct, then intervening on those internals should change behavior in the predicted way. That testability is what separates the field from loose attempts to read meaning into a heat map.

How it differs from other "explainable AI" methods

Older explainability tools, such as saliency maps and attribution scores, ask which parts of the input mattered most for a given output. They are useful and cheap, but they describe the relationship between input and output, not the computation between them. Mechanistic work tries to open the middle.

Approach	Question it answers	Typical output	Limitation
Behavioral evaluation	What does the model do on these tests?	Accuracy scores, failure cases	Silent about internal reasons
Input attribution (saliency)	Which input tokens or pixels mattered?	Heat map over the input	Does not describe the internal algorithm
Probing	Is some concept linearly decodable from a layer?	Classifier accuracy	Decodable does not mean used
Mechanistic interpretability	Which internal components compute this, and how?	Features, circuits, causal tests	Labor-intensive and partial

The probing row deserves a note, because it is the closest relative. A probe trains a small classifier to read, say, grammatical number from a layer's activations. If the classifier succeeds, the information is present. It does not show that the model uses that information downstream. Mechanistic methods add the causal test: change the internal state and see whether the output moves.

Features, neurons, and why neurons are the wrong unit

The obvious first guess is that each neuron in a network stands for one concept: a "cat" neuron, a "legal text" neuron. For some neurons in some vision networks, something like that holds. A 2020 Distill article led by Chris Olah, "Zoom In: An Introduction to Circuits," studied the InceptionV1 image model and described neurons that detect curves at particular orientations, which is the kind of tidy finding that makes the cell-biology comparison appealing. Biologists look at cells under a microscope; interpretability researchers look at individual units and the connections between them.

The tidy picture breaks down quickly. Many neurons respond to several unrelated things. The same article describes polysemantic neurons (neurons that respond to multiple unrelated inputs), including one that fired for both cat faces and car fronts. Polysemantic is the technical word, and its opposite, monosemantic, describes a unit that responds to one coherent concept.

Researchers use the word feature for the concept itself, whether or not any single neuron carries it. A feature is a direction of variation the network treats as meaningful: "this text is in French," "this token is inside a code comment," "the subject of this sentence is plural." The working hypothesis is that networks compute by detecting and combining features, and that features are the natural unit of analysis, not neurons.

Superposition: more ideas than slots

If features are not neurons, where do they live? The answer from Anthropic's 2022 paper "Toy Models of Superposition" (Elhage and colleagues) is that a network can store more features than it has dimensions by packing them in overlapping, nearly orthogonal directions. The paper trained small ReLU networks on synthetic data with sparse input features and showed when models represent more features than they have dimensions. The authors call this superposition, and they argue it is a main cause of polysemantic neurons.

An analogy helps here. Imagine a hotel with 512 rooms and a guest list of 4,000 people, where on any given night only a handful of guests show up. You can assign each guest a small group of rooms, and as long as guests rarely arrive together, the occupied rooms usually identify who is there. Collisions happen occasionally and cause confusion, and the network needs ways to filter that interference. That trade is what the paper describes: sparsity makes the compression workable, at the cost of interference that has to be cleaned up nonlinearly.

The practical consequence is uncomfortable. If a model really does hold far more features than neurons, then reading neurons one at a time will always yield a muddle, however carefully you do it. You need a method that unmixes the overlap.

The authors of the superposition paper float a larger idea: that practical neural networks may be noisily simulating much larger, highly sparse networks. If so, the neurons we can see are a compressed shadow of a bigger, cleaner computation.

Circuits: how features pass information along

A feature on its own is a detector. A circuit is a connected set of features and the weighted connections between them that together carry out some piece of computation. The Zoom In article defined circuits as subgraphs of a network: a set of features plus the edges between them. In an image model, simple detectors such as curve detectors feed later units that respond to more complex shapes, and each link can be inspected.

For transformers, the setup is somewhat different from the image networks that started the field. A December 2021 Anthropic paper, "A Mathematical Framework for Transformer Circuits" by Nelson Elhage and colleagues, laid out a way of thinking about them. The central idea is the residual stream: a running vector that each component reads from and writes back to. The paper describes components as communicating by reading and writing to different subspaces of that stream, a bit like colleagues working on a shared whiteboard where each person only looks at the corner relevant to them.

That paper analyzed deliberately small models: zero-layer, one-layer, and two-layer transformers with attention only and no MLP layers (the feed-forward blocks). The authors say plainly that they had much less success interpreting MLP layers, and called this a major limitation. Keep that caveat in mind, because it recurs through the whole field: the cleanest results come from the smallest and simplest systems.

Induction heads: the first widely replicated circuit

The best-known transformer circuit is the induction head. The idea is simple. Suppose the text so far contains the pattern A followed by B somewhere earlier, and now A appears again. An induction head looks back, finds the earlier A, notices that B followed it, and makes B more likely to come next. Written schematically, it completes [A][B] ... [A] with [B].

It takes two attention heads in different layers working together. One, a previous-token head, copies information about the preceding token into each position. The second, the induction head proper, uses that information to find earlier matches and attend to what came after them. Neither head does anything impressive alone; the circuit appears only when both are present in the right layers.

The 2022 paper "In-context Learning and Induction Heads" (Olsson and colleagues, March 8, 2022) tied this to a bigger question: how do language models learn from examples placed in their prompt? The authors found that during training, models with more than one layer pass through a phase change, a brief bump in the training loss, during which most in-context learning ability forms and induction heads form at the same time. They argued that induction heads are plausibly responsible for much of in-context learning.

The paper is open about how strong the evidence is. It describes the case as circumstantial and says it is stronger for small models than for large ones. For large models the claim leans mainly on continuity with the small ones. That honesty is worth noticing, since popular summaries tend to flatten it into "induction heads explain in-context learning."

A natural-language circuit in GPT-2 small

Induction heads were found in toy models and then looked for in real ones. A 2022 paper by Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt, "Interpretability in the Wild," took a natural language task in GPT-2 small, indirect object identification, and traced it to a circuit. In a sentence like "When Mary and John went to the store, John gave a drink to ___," the model needs to output "Mary": the person who is not the subject that appeared twice.

The authors report that the explanation involves 26 attention heads, grouped into seven main classes, found with causal interventions: swapping or ablating a component and checking whether the behavior changes. They evaluated the account on three criteria (faithfulness, completeness, and minimality) and presented it as the largest end-to-end reverse-engineering of a natural behavior in a language model at the time. Twenty-six heads for one grammatical trick, in the smallest GPT-2, gives a sense of how much machinery hides behind even a modest behavior.

Sparse autoencoders: unmixing the superposition

If superposition scrambles features across neurons, the fix is a tool that unscrambles them. The leading tool is the sparse autoencoder, a small neural network trained on a larger model's internal activations. It has two jobs that pull against each other. It must reconstruct the activations accurately, and it must do so using only a few active "dictionary" entries at a time. The sparsity pressure pushes each entry toward representing one clean concept. The method is also called dictionary learning, because the learned entries form a dictionary from which any activation can be assembled as a sparse sum.

A useful comparison is a prism. White light looks like a single thing until a prism spreads it into component colors. The sparse autoencoder is meant to do the same with an activation vector: take the overlapping mixture and spread it into the features that were summed to make it.

Towards Monosemanticity (October 2023)

Anthropic's "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning" applied this to a one-layer transformer. Anthropic's own summary says the 512-neuron layer was decomposed into more than 4,000 features, covering things like DNA sequences, legal language, HTTP requests, Hebrew text, and nutrition statements. The features, the authors argued, were more interpretable than the neurons they came from. Importantly, they were patterns across several neurons, not single neurons, which is the whole point: the unit of meaning had moved.

One-layer models are very far from a deployed chatbot, so the obvious question was whether it would scale.

Scaling to a production model (May 2024)

On May 21, 2024, Anthropic published "Mapping the Mind of a Large Language Model," extracting millions of features from the middle layer of Claude 3 Sonnet. The features were often abstract and multilingual. A feature for the Golden Gate Bridge activated on English mentions of the bridge, on discussions in Japanese, Chinese, Greek, Vietnamese, and Russian, and on images of it. Nearby features included Alcatraz and the 1906 San Francisco earthquake, so the geometry of features reflected conceptual closeness.

Because the method yields causal handles, the team could also turn features up. When the Golden Gate Bridge feature was amplified, Claude answered a question about its physical form by claiming to be the bridge and kept bringing it up in unrelated answers. The scientific value of the demonstration was in showing that the feature was causally connected to behavior, not just correlated with it.

Other features were more sobering. The team described features related to code backdoors, bioweapons, gender discrimination, power-seeking, manipulation, and secrecy. A scam-email feature, when strongly activated artificially, led Claude to draft a scam email despite its usual refusal. A sycophancy-related feature, amplified, made Claude respond to an overconfident user with fawning, untruthful praise.

The authors listed their own limits. The features found were a small subset of all the concepts the model learned. Finding a full set with current techniques would be cost-prohibitive, because the computation would exceed the compute used to train the model in the first place. And knowing a feature exists does not show how the model uses it: the circuits that involve each feature still need to be found.

Case studies: algorithms the network invented

Circuits and features are abstractions. The most convincing evidence that the field is doing something real comes from cases where researchers recovered a clean, human-checkable algorithm from the weights.

Modular addition and grokking

Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt studied small transformers trained on modular addition, adding two numbers and reporting the remainder after dividing by a fixed number. In their paper "Progress measures for grokking via mechanistic interpretability" (arXiv, January 2023), they reverse-engineered the learned algorithm. It converts addition into rotation around a circle, using discrete Fourier transforms and trigonometric identities. The authors verified this by analyzing activations and weights and by ablating components in Fourier space.

Think of a clock face. Adding 5 hours to 9 o'clock gives 2 o'clock, because the hands wrap around. The network discovered, on its own, a version of the clock: it represents numbers as angles and adds by combining rotations. No one told it to do this.

The paper uses the discovery to explain grokking, a puzzling pattern where a network first memorizes its training data, performs poorly on new data for a long stretch, then abruptly generalizes. With progress measures derived from the circuit, the authors divided training into three phases: memorization, circuit formation, and cleanup. Their conclusion was that grokking is a gradual strengthening of structured mechanisms followed by removal of the memorizing components, not a sudden jump. The sudden-looking jump in test accuracy hides smooth progress that only becomes visible once you know what to measure.

Tracing thoughts in Claude 3.5 Haiku (March 2025)

On March 27, 2025, Anthropic published "Tracing the thoughts of a large language model," using tools that follow chains of features through a model on specific prompts. The work studied Claude 3.5 Haiku, and its findings read as small, concrete surprises:

• Shared concepts across languages. Internal features for English, French, and Chinese overlapped, suggesting a common conceptual space before the answer is put into a specific language. The overlap grew with model scale.
• Planning a rhyme. When asked to write a rhyming second line of a poem, the model picked a rhyme word in advance and built the line toward it. Suppressing that word led to a different planned rhyme, and injecting a new concept changed the ending again.
• Parallel arithmetic paths. For a sum like 36 plus 59, one path estimated the approximate total while another determined the final digit. Asked how it did the sum, the model described the standard carry-the-one method, not what the trace showed.
• Two-step recall. For "the capital of the state containing Dallas," the model activated an intermediate fact (Dallas is in Texas) before reaching Austin. Swapping the Texas concept for California changed the answer to Sacramento.

The arithmetic result matters beyond curiosity. It shows a case where a model's explanation of its own process differs from its measured process. That gap is exactly the situation mechanistic methods exist to detect, and it is a reason to be careful about taking a chatbot's self-report as evidence of how it reached an answer.

What interpretability cannot yet do

The results above are real, and also narrow. Several limits come from the researchers themselves.

Coverage is partial. The Anthropic team states that even on short prompts their method captures only part of the computation. Some mechanisms they observed may be artifacts of the analysis tools rather than features of the model. The Monosemanticity and Claude 3 Sonnet work found subsets of features, not complete inventories.

It is slow and human-intensive. Interpreting the circuits behind a prompt of only tens of words took several hours of human effort in the 2025 study, so scaling to the long, multi-step reasoning chains that modern assistants produce will require better methods. For comparison, the indirect-object circuit in GPT-2 small, a model far smaller than current systems, required characterizing 26 heads.

Features are not yet circuits. Knowing the features is half the job. Anthropic's 2024 write-up says directly that knowing what features exist does not show how the model uses them.

Early tools worked on simplified models. The 2021 framework paper studied attention-only transformers with no MLP layers, and said MLP layers were much harder. The induction-head paper called its evidence stronger for small models than for large ones.

Safety use is unproven. Finding a "deception" or "sycophancy" feature is not the same as demonstrating that monitoring or steering it makes a deployed system safer. Anthropic's 2024 page says the safety-relevant features found so far still need to be shown to improve safety.

None of this means the field is stalled. It means the claims have to be matched to the evidence: a verified circuit in a small model is a different kind of result from a feature dictionary in a production model, which is different again from a complete understanding of a frontier system, which no one has.

Why it matters for safety claims

The reason mechanistic work attracts funding and attention is its connection to AI safety and alignment. Techniques like those in RLHF, RLAIF, and DPO shape what a model outputs. They do not guarantee anything about why. Two models can give the same polite answer for different internal reasons, one of them fragile or deceptive, and output-level testing cannot always tell them apart.

Interpretability offers a different kind of check. In principle you could verify that a model is not planning something it does not say, or detect that an apparently good answer came from a spurious shortcut. The 2025 poem result is a small example of the principle: output inspection would show only a rhyming line, while the internal trace showed the rhyme being chosen before the line was written.

Interpretability also bears on everyday failure modes. The reasons a model invents facts are still debated, and circuit-level studies are one route to separating "the model does not know" from "the model failed to retrieve what it knows." The mixture-of-experts design described in mixture of experts routing adds another layer to inspect, since different experts may specialize in different features. And understanding what the layers of a neural network store, and how attention moves information between positions, is the groundwork for every circuit result above.

The most careful way to state the safety case is conditional. If researchers can reliably identify the features relevant to a risk, and show that intervening on them changes behavior as predicted, then interpretability could become one input to safety evaluations alongside behavioral tests. That "if" is still open, and responsible summaries should say so.

Common misconceptions

"Interpretability means the model explains itself." No. Asking a chatbot to explain its reasoning produces more generated text, and the arithmetic example shows that text can describe a different method from the one the model used. Mechanistic interpretability inspects the computation, not the narration.

"Researchers have decoded the model." They have decoded pieces of specific models on specific tasks. The authors of the major papers describe their coverage as partial.

"A feature is a neuron." The field's central move is the observation that it usually is not. Features are directions spread across neurons, which is why neuron-by-neuron inspection gets muddled.

"Amplifying a feature proves it is the cause of a behavior." It shows the feature is sufficient to nudge behavior in that setting. Whether the feature is what the model normally relies on is a separate question, which is why researchers also run ablations (turning things off) and check both directions.

"Smaller toy models tell us nothing." They tell us what to look for. Induction heads were found in two-layer models first; the grokking algorithm was found in a small transformer first. The open question is how much transfers to large models, and the evidence for large models is generally weaker.

Frequently asked questions

Q: What is mechanistic interpretability in one sentence? A: It is the study of how a trained neural network computes its outputs, by identifying internal features and the circuits connecting them and testing those explanations with causal interventions such as ablating or amplifying a component.

Q: What is the difference between a feature and a circuit? A: A feature is a concept or pattern the network represents internally, such as text being in French. A circuit is a set of features and the weighted connections between them that together perform a computation, such as an induction head completing a repeated pattern.

Q: What is superposition in a neural network? A: Superposition is a network storing more features than it has dimensions by overlapping them in nearly orthogonal directions. Anthropic's 2022 toy-model paper argues it works because most features are sparse, meaning rarely active together, and that it is a main cause of polysemantic neurons.

Q: What is a sparse autoencoder used for in interpretability? A: It is a small network trained to reconstruct a larger model's internal activations using only a few active dictionary entries at a time. The sparsity pressure encourages each entry to correspond to a single concept, which unmixes features that superposition had overlapped across neurons.

Q: Has anyone fully reverse-engineered a large language model? A: No. Published work has reverse-engineered small models, single behaviors such as indirect object identification in GPT-2 small, and a subset of features in larger models. Anthropic's own 2025 study states its method captures only part of the computation even on short prompts.

Q: Why did Claude become obsessed with the Golden Gate Bridge in the 2024 demonstration? A: Researchers had found a feature in Claude 3 Sonnet that fires on mentions and images of the bridge. When they artificially amplified it, the model brought the bridge into unrelated answers and even claimed to be the bridge, showing the feature was causally connected to behavior.

Q: Does interpretability make AI safe? A: Not by itself. It may help detect risky internal features or shortcuts that output testing misses, but the 2024 Anthropic work says safety-relevant features still need to be shown to improve safety in practice.

The takeaway

The core insight of mechanistic interpretability is that a neural network's meaning does not sit in its individual neurons. It sits in overlapping directions that can be pulled apart with the right tools, and in the circuits that connect them, some of which look like recognizable algorithms: a copy-the-pattern routine, a clock-face method for addition, a rhyme chosen before the line is written. Each of those results was checked by intervening on the model, which is what makes them more than stories.

The honest scorecard is mixed. The method has produced verified mechanisms in small systems and a large catalog of features in a production model, and it has also produced a clear list of what remains out of reach: complete coverage, speed, and proof that any of it improves safety. Treat any claim that a large model has been "decoded" with the same skepticism the researchers apply to their own work.

Sources

• Anthropic, "Tracing the thoughts of a large language model," March 27, 2025: https://www.anthropic.com/research/tracing-thoughts-language-model
• Anthropic, "Mapping the Mind of a Large Language Model," May 21, 2024: https://www.anthropic.com/research/mapping-mind-language-model
• Olsson et al., "In-context Learning and Induction Heads," transformer-circuits.pub, March 8, 2022: https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html
• Elhage et al., "A Mathematical Framework for Transformer Circuits," transformer-circuits.pub, December 22, 2021: https://transformer-circuits.pub/2021/framework/index.html
• Nanda et al., "Progress measures for grokking via mechanistic interpretability," arXiv:2301.05217: https://arxiv.org/abs/2301.05217
• Wang et al., "Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small," arXiv:2211.00593: https://arxiv.org/abs/2211.00593
• Elhage et al., "Toy Models of Superposition," arXiv:2209.10652: https://arxiv.org/abs/2209.10652
• Anthropic, "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning" (October 2023): https://www.anthropic.com/research/towards-monosemanticity-decomposing-language-models-with-dictionary-learning
• Olah et al., "Zoom In: An Introduction to Circuits," Distill, 2020: https://distill.pub/2020/circuits/zoom-in/

For wider context on where this fits among AI topics, see the complete AI guide, and for how vector representations work in practice see embeddings explained.