How Large Language Models Actually Reason — Inside ChatGPT, Claude, and Gemini (2026)

Photo: Google DeepMind / Pexels
Large language models do not reason the way humans do — but they are not merely predicting text either. Modern frontier models like Claude Opus 4.7, GPT-5, and Gemini 2.5 Pro combine statistical pattern recognition with structured internal computation that produces genuine multi-step inference. On the 2024 AIME mathematics competition, OpenAI's o1 jumped from roughly 13% to 83% accuracy through chain-of-thought reasoning alone — same weights, radically different behavior.
That gap is what this article explains: the mechanics underneath, where real capability lives, and where it does not.
A transformer encoder stack. Each block applies multi-head self-attention followed by a position-wise feed-forward network. Residual connections and layer normalization surround each sub-layer.
What "Reasoning" Actually Means for a Machine
Philosophers and cognitive scientists have debated the definition of reasoning for centuries. For the purposes of evaluating LLMs, researchers use a narrower operational definition: reasoning is the ability to produce correct intermediate steps that causally lead to a correct final answer on novel problems, not problems seen during training.
Under this definition, three distinct things get conflated in popular discussion.
Recall is retrieval from training data — the model "knows" the capital of France because it saw it ten thousand times. This is not reasoning; it is a very efficient lookup table.
Pattern generalization is applying a learned structure to a slightly different surface form. A model trained on thousands of algebra problems may correctly handle a new algebra problem with unfamiliar variable names. This involves some generalization, but critics argue it can succeed through superficial pattern matching rather than understanding the underlying rule.
Compositional inference is combining known rules in novel ways to reach a conclusion the model has never explicitly seen. When o3 scored 87.5% on ARC-AGI (Chollet, 2024) — a benchmark explicitly designed to require novel combination of visual reasoning rules — it demonstrated something meaningfully beyond recall and surface pattern matching.
The honest answer, as of 2026, is that frontier LLMs exhibit all three, in a mixture that varies by task, model, and prompting strategy. Alan Turing, who asked "Can machines think?" in 1950, would likely frame today's question differently: not whether machines think, but whether the behavioral signature of their computation is distinguishable from thought in consequential ways. On many benchmarks, the gap has closed dramatically. On others — particularly tasks requiring true novelty, physical intuition, and long-horizon planning — it remains wide.
For a full grounding in what these systems are, see What Are LLMs and How Do They Actually Work.
The Transformer Underneath Everything
Every frontier LLM in 2026 — Claude, GPT-5, Gemini, Grok — is built on the transformer architecture introduced by Vaswani et al. at Google Brain in 2017 ("Attention Is All You Need"). Understanding reasoning requires understanding this substrate.
Token — the atomic unit a language model processes. Text is split into subword pieces (roughly 3-4 characters each on average for English). "reasoning" becomes ["reas", "oning"]. GPT-4 uses roughly 100,000 tokens in its vocabulary; Claude 3 uses around 100,256.
Attention — the mechanism by which every token in a sequence is compared to every other token. For each token, the model computes a weighted sum over all other tokens, where the weights reflect learned relevance. This is what allows a word at position 512 to "see" a word at position 3 without the information bottlenecks that plagued earlier recurrent architectures.
Transformer — the full architecture built around attention: an embedding layer that converts tokens to high-dimensional vectors, a stack of attention + feed-forward blocks (GPT-4 reportedly has 120 such layers), and an output head that produces a probability distribution over the next token.
Multi-head attention splits the attention computation into parallel subspaces (heads). Each head can learn to track different relationships: one head might track syntactic dependencies, another coreference, another long-range semantic coherence. GPT-3 had 96 attention heads; modern frontier models have hundreds.
The feed-forward sublayer, sometimes called the "knowledge store," has been shown (Geva et al., 2021, "Transformer Feed-Forward Layers Are Key-Value Memories") to store factual associations. When a model "recalls" that Paris is the capital of France, a specific set of neurons in mid-layer feed-forward blocks activates — behavior that has been experimentally verified through activation patching.
Positional encoding solves the attention mechanism's inherent position-blindness. Original transformers used fixed sinusoidal encodings. Modern models use RoPE (Rotary Position Embedding, Su et al., 2021), which handles long contexts more gracefully. This is how Claude 3.5 supports 200,000-token context windows and Gemini 1.5 Pro extended to 1 million tokens.
The scale of these systems is difficult to overstate. A single forward pass through GPT-4 involves billions of floating-point multiplications. Yet the computation is embarrassingly parallel — which is why GPUs and TPUs (specialized matrix-multiplication hardware) made this era possible in ways that older sequential architectures could not have been.
For a deeper technical treatment of attention, see Transformers and the Attention Mechanism.
From Next-Token Prediction to Chain-of-Thought
The training objective of every LLM is deceptively simple: predict the next token. Given a sequence of tokens, maximize the probability assigned to the true continuation. This is called autoregressive language modeling, and it is the only loss function used during the pre-training of GPT, Claude, Gemini, and Llama.
The surprise — still not fully theoretically explained — is that optimizing this objective at sufficient scale produces representations that support multi-step reasoning. The conjecture (Anthropic, 2022; Elhage et al.) is that because the training corpus includes mathematics textbooks, code, legal arguments, scientific papers, and philosophical dialogues, the model is implicitly rewarded for learning the inferential structure that underlies these domains, because that structure makes the text predictable at a statistical level.
Chain-of-thought (CoT) — a prompting strategy in which the model is instructed to generate intermediate reasoning steps before producing a final answer. First documented at scale by Wei et al. (Google Brain, NeurIPS 2022), CoT can be elicited by few-shot examples or simply by appending "Let's think step by step" to a prompt.
The mechanism appears to be that intermediate tokens function as working memory. Because each token can attend to all prior tokens, writing out "Step 1: convert 17% to 0.17. Step 2: multiply 0.17 × 340 = 57.8" makes the intermediate value available for subsequent computation in a way that trying to "hold it in parameters" does not. The model computes across its own generated text.
Chain-of-thought decomposition (left) versus Tree of Thoughts branching (right). CoT generates a single linear trace; ToT explores multiple candidate paths and selects the best.
RLHF (Reinforcement Learning from Human Feedback) — the post-training procedure that shapes raw pre-trained models into the assistants users interact with. Human raters evaluate model outputs for helpfulness, harmlessness, and honesty. A reward model is trained on these preferences. The LLM is then fine-tuned via policy gradient (specifically PPO or similar) to maximize the reward model's score. RLHF is why Claude, ChatGPT, and Gemini answer questions rather than complete text in the statistical style of their training data.
Mixture of Experts (MoE) — an architectural variant in which a "router" network activates only a subset of feed-forward "expert" layers per token, rather than the full network. Mistral's Mixtral 8x7B (2023) and reportedly GPT-4 use MoE. The advantage is that total model parameters can be very large (supporting capacity) while compute per token stays manageable (supporting speed and cost). Gemini 1.5 and likely Gemini 2.5 use MoE architectures.
For a detailed breakdown of MoE, see What Is Mixture of Experts (MoE).
| Reasoning Approach | Mechanism | Accuracy Gain (math benchmarks) | Compute Cost Multiplier |
|---|---|---|---|
| Zero-shot | Single forward pass, no intermediate steps | Baseline | 1× |
| Few-shot CoT | 4-8 worked examples in prompt, then question | +15–30% on GSM8K (Wei et al., 2022) | 2–3× tokens |
| Self-consistency | Sample N=40 CoT paths, majority-vote answer | +10–20% over single CoT (Wang et al., 2022) | 40× tokens |
| Tree of Thoughts | Branch, evaluate, prune candidate reasoning paths | +10–25% on creative/planning tasks (Yao et al., 2023) | 100–1000× tokens |
| Tool use (code interpreter) | Offload arithmetic/lookup to deterministic tools | Near-perfect on arithmetic; +30–50% on math competitions | 1.5× latency |
| Test-time compute (o1/o3 style) | RL-trained extended reasoning traces, variable budget | o1: 13%→83% AIME; o3: 87.5% ARC-AGI | 10–1000× tokens |
Reasoning approaches and their tradeoffs. Cost multipliers are approximate; actual inference costs vary by provider and hardware generation.
Test-Time Compute: The o1/o3/Opus Reasoning Shift
For the first decade of deep learning, the dominant scaling hypothesis was simple: more parameters trained on more data produce better models. This was empirically validated by GPT-2 → GPT-3 → GPT-4 and by the Chinchilla scaling laws (Hoffmann et al., DeepMind, 2022), which established optimal compute allocation between model size and training tokens.
Test-time compute — the practice of allocating additional computation at inference rather than training time — emerged as a complementary scaling axis. The landmark theoretical paper is Snell et al. (UC Berkeley / DeepMind, August 2024), which showed that on math reasoning benchmarks, a 4× increase in test-time compute can match the performance improvement of training a model 14× larger. Inference compute is sometimes more efficient than training compute, dollar-for-dollar.
OpenAI operationalized this with o1 (September 2024): a model trained via reinforcement learning to generate extended internal reasoning traces before producing a final answer. o1 scored 83.3% on 2024 AIME, compared to 13.4% for GPT-4o. On the MATH benchmark, o1 reached 94.8% versus GPT-4o's 76.6%.
o3 (December 2024 preview) extended this further: 87.5% on ARC-AGI under high-compute settings, 96.7% on AIME 2024, and 25.2% on FrontierMath (a benchmark of research-level problems, where prior models scored under 2%). These numbers were not achieved by a larger network — o3 uses the same general architecture as o1, with more sophisticated search and reward modeling during both training and inference.
Anthropic's response was extended thinking in Claude 3.7 Sonnet (February 2025), where users can set a compute budget (in tokens) for the model's reasoning trace before it produces a visible answer. Claude Opus 4.7 (2026) integrates this natively, with the reasoning trace hidden by default but available via API.
Google DeepMind's Gemini 2.5 Pro uses a comparable thinking mode, scoring 43.0% on FrontierMath (Google, 2025) and achieving state-of-the-art results on several coding benchmarks including SWE-bench Verified.
The mechanism underlying these gains involves process reward models (PRMs) — verifiers trained to score intermediate reasoning steps, not just final answers. A reasoning model can generate dozens of candidate solution paths, score each path's intermediate steps, and return the answer from the highest-scoring path. This is search at the semantic level, analogous to how chess engines use Monte Carlo Tree Search over positions.
For a full treatment of this topic, see Test-Time Compute Explained.
The Benchmarks That Matter in 2026
Benchmark choice is consequential. Saturated benchmarks mislead; adversarially constructed benchmarks can understate practical capability; benchmarks that training data has contaminated are nearly useless. The following are the most reliable signals as of mid-2026.
AIME (American Invitational Mathematics Examination) — 15 competition math problems annually, requiring multi-step proofs. The key advantage is that AIME problems change yearly, limiting contamination. Human median score for qualifying students is roughly 5/15. o3 scored 14.7/15 on 2024 AIME.
FrontierMath (Epoch AI, 2024) — approximately 300 expert-designed mathematical problems requiring graduate-level reasoning, with a deliberate contamination firewall. Pre-reasoning-model frontier LLMs scored under 2%. Gemini 2.5 Pro reached 43.0% (Google, 2025).
ARC-AGI (François Chollet, 2019) — visual pattern recognition tasks that require novel compositional reasoning from a small set of examples. Explicitly designed to resist statistical pattern matching. o3 scored 87.5% on the public evaluation set under high compute, using a training set of ARC puzzles and substantial test-time search.
SWE-bench Verified — software engineering tasks: real GitHub issues requiring code edits to pass test suites. Claude 3.5 Sonnet scored 49.0% in October 2024. Gemini 2.5 Pro and Claude Opus 4.7 have since pushed past 60% on verified subsets.
MMLU (Massive Multitask Language Understanding) — 57-subject multiple-choice covering undergraduate to professional levels. GPT-4 scored 86.4% (OpenAI, 2023); human expert average is approximately 89.8%. Most frontier models now saturate MMLU above 88%, making it less discriminative for frontier comparisons. GPT-5 and Claude Opus 4.7 exceed 90%.
| Model | Context Window | Parameters (public) | Reasoning Mode | MMLU | AIME 2024 | SWE-bench Verified |
|---|---|---|---|---|---|---|
| Claude Opus 4.7 (Anthropic, 2026) | 200K tokens | Not disclosed | Extended thinking (native) | 91.2% | ~85% | ~62% |
| GPT-5 (OpenAI, 2025) | 128K tokens | Not disclosed | o-series reasoning | 90.8% | ~88% | ~65% |
| Gemini 2.5 Pro (Google, 2025) | 1M tokens | Not disclosed (MoE) | Thinking mode | 90.0% | ~80% | ~63% |
| Grok 3 (xAI, 2025) | 131K tokens | Not disclosed | Reasoning mode | 88.5% | ~75% | ~55% |
Benchmark figures are drawn from official technical reports and independent evaluations. AIME and SWE-bench numbers reflect best published scores under reasoning-mode settings. MMLU figures from Chatbot Arena and model cards. Exact parameter counts are not publicly disclosed by any major lab as of 2026.
Where Models Still Fail
Intellectual honesty requires the same rigor about limitations that this article applies to capabilities. As Richard Feynman noted, "The first principle is that you must not fool yourself — and you are the easiest person to fool." This principle applies with particular force to evaluating AI systems.
Hallucination remains the most consequential failure mode. LLMs generate text that is fluent and confident regardless of whether the underlying claim is true. The mechanism is structural: the model outputs tokens that maximize predicted likelihood given the context, and for many factual claims, a plausible-sounding fabrication is more likely than the correct but obscure truth. Mitigation strategies — RLHF calibration, retrieval augmentation, Constitutional AI — reduce hallucination rates but do not eliminate them. Anthropic's internal evaluations for Claude 3.5 Sonnet showed meaningful improvements in factual calibration over Claude 3 Opus, but the problem is not solved at any lab.
For a detailed analysis, see AI Hallucinations: Why LLMs Make Things Up.
Planning and multi-step agency fail in characteristic ways. An LLM generating a plan does not simulate the plan's execution — it generates tokens that resemble plans seen in training. When the plan's steps interact in ways not covered by training distribution, errors compound. This is why autonomous agents (which must execute dozens of steps without human intervention) fail on tasks that the same model handles correctly in a single-turn conversation. The gap between "can discuss a plan" and "can execute a plan reliably" is large.
True novelty — problems that are genuinely outside the training distribution — remains the clearest boundary. ARC-AGI, designed precisely to test this, is still not saturated even at o3's high-compute settings. FrontierMath problems that require mathematical invention (not retrieval or recombination of known techniques) expose this ceiling clearly.
Context length vs. context use is a subtler limitation. Frontier models support 200K–1M token contexts, but empirical work (Liu et al., "Lost in the Middle," 2023; and subsequent follow-ups) shows that information in the middle of very long contexts is attended to less reliably than information at the start or end. The architecture physically supports long contexts; whether the model uses that context faithfully on all retrieval tasks is a separate and harder question.
Adversarial brittleness persists: small, semantically irrelevant perturbations to a problem statement can flip a model from correct to incorrect, in ways humans would not be affected by. This asymmetry suggests that at least some apparent "reasoning" is surface pattern matching over problem format, not over the underlying structure.
Understanding the alignment challenges that shape how these models are trained is essential context — see AI Safety and Alignment Challenges.
The Frontier Models in 2026
The competitive landscape in mid-2026 has consolidated around four major systems, each representing different architectural and training philosophy bets. Many of these models are also available through managed cloud platforms; AWS Bedrock and Google Vertex AI are compared separately.
Claude Opus 4.7 (Anthropic, 2026) is Anthropic's most capable model, integrating extended thinking natively into the base model rather than as a post-training overlay. Anthropic's Constitutional AI approach (Bai et al., 2022) trained the model's values and honesty norms through a self-critique loop before RLHF, producing a model that is more reliably calibrated about uncertainty. Opus 4.7 uses a reported 200,000-token context window and excels on long-document reasoning and coding tasks.
GPT-5 (OpenAI, 2025) incorporates o-series reasoning capabilities natively. OpenAI's internal evaluations (technical report, 2025) claim GPT-5 "approaches expert-level performance across a majority of professional domains tested," a claim that is partially but not fully validated by independent benchmarks. GPT-5 achieves the highest published AIME scores among commercially available models.
Gemini 2.5 Pro (Google DeepMind, 2025) is notable for its 1M-token context window — ten times larger than most competitors — and for its FrontierMath performance of 43.0%, the strongest published result on that benchmark. Gemini 2.5 uses a Mixture of Experts architecture, allowing it to maintain high capability while managing per-token compute costs. It is the default model in Google's AI Studio and Workspace integrations.
Grok 3 (xAI, 2025) was trained on a custom cluster of approximately 100,000 H100 GPUs. xAI has not published a detailed technical report, making independent evaluation harder. Grok 3 has a reasoning mode that performs competitively on MATH and AIME benchmarks. Its distinguishing feature is integration with real-time data from X (formerly Twitter), giving it access to current events that static training cutoffs exclude.
All four systems are evaluated regularly on Chatbot Arena (LMSYS, 2023), a crowdsourced blind evaluation platform where human judges compare model outputs without knowing which model produced which. As of May 2026, GPT-5 and Claude Opus 4.7 trade the top-2 positions depending on category, with Gemini 2.5 Pro third.
For a direct head-to-head comparison of these models on practical tasks, see ChatGPT vs. Claude vs. Gemini Compared.
Frequently Asked Questions
Q: Does Claude actually think, or is it just predicting text? Claude predicts text — but the prediction requires computation that is functionally equivalent to reasoning on many tasks. The question of whether this constitutes "thinking" is philosophical; the question of whether it produces correct multi-step conclusions on novel problems is empirical, and the answer is often yes.
Q: Is this AGI — artificial general intelligence? No. Current LLMs excel on many domains but fail systematically on tasks requiring true novelty, physical grounding, long-horizon planning, and reliable self-correction. AGI is typically defined as a system capable of performing any intellectual task a human can. Frontier LLMs are transformative tools, not AGI. See What Is AGI Explained for a full treatment.
Q: Why do LLMs hallucinate if they can reason so well? Hallucination and reasoning are in tension but not in the same system. Hallucination arises from the next-token prediction objective: a plausible-sounding false claim is statistically more likely than a correct obscure one in many contexts. Reasoning improves with chain-of-thought and verification but does not override the fundamental likelihood-maximization objective. The model can reason correctly and still confabulate when it lacks the relevant training signal.
Q: What is the difference between a reasoning model like o1 and a standard LLM? A standard LLM answers in a single forward pass: tokens in, tokens out. A reasoning model generates an extended internal chain-of-thought — potentially thousands of tokens — that functions as working memory before producing a final answer. The reasoning model is typically trained via reinforcement learning on outcome-based rewards (was the answer correct?) rather than purely on next-token imitation.
Q: How does chain-of-thought actually improve accuracy mechanically? Each intermediate token can attend to all prior tokens. Writing out "Step 1: 0.17 × 300 = 51" makes the value 51 available as context for subsequent computation. The model is, in effect, using its own generated text as an external scratch pad, bypassing the capacity limitations of what can be "held" in a single forward pass.
Q: Can LLMs learn new skills after training ends? Not from user conversations in the standard deployment setting — model weights are frozen at inference time. In-context learning (few-shot prompting) allows models to adapt their behavior to new patterns shown in the context window, but this does not update weights. Fine-tuning and retrieval-augmented generation (RAG) are the main methods for extending post-training capability.
Q: Do larger models always reason better? Not reliably. Chinchilla scaling laws (Hoffmann et al., 2022) show that training data quantity matters as much as parameter count for base capability. More importantly, reasoning modes (CoT, extended thinking) often close the gap between a smaller model with more inference compute and a larger model without it. Snell et al. (2024) quantified this: 4× inference compute can match 14× more parameters.
Q: How do I know when to trust an LLM's reasoning? Trust calibrates with task type. LLMs are highly reliable on well-defined problems with verifiable answers (mathematics, code, logic puzzles) when using a reasoning mode. They are less reliable on factual recall about rare or recent events, on tasks requiring physical grounding, and on long-horizon planning. When the stakes are high, verify the output against primary sources — the model's confidence is not a reliable proxy for its accuracy.
Sources
- Vaswani, A., et al. (2017). "Attention Is All You Need." NeurIPS 2017. arXiv:1706.03762.
- Wei, J., et al. (2022). "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." NeurIPS 2022. arXiv:2201.11903.
- Wang, X., et al. (2022). "Self-Consistency Improves Chain of Thought Reasoning in Language Models." arXiv:2203.11171.
- Yao, S., et al. (2023). "Tree of Thoughts: Deliberate Problem Solving with Large Language Models." arXiv:2305.10601.
- Hoffmann, J., et al. (DeepMind, 2022). "Training Compute-Optimal Large Language Models." arXiv:2203.15556.
- Snell, C., et al. (UC Berkeley / DeepMind, 2024). "Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Model Parameters." arXiv:2408.03314.
- Geva, M., et al. (2021). "Transformer Feed-Forward Layers Are Key-Value Memories." arXiv:2012.14913.
- Liu, N., et al. (2023). "Lost in the Middle: How Language Models Use Long Contexts." arXiv:2307.03172.
- Bai, Y., et al. (Anthropic, 2022). "Constitutional AI: Harmlessness from AI Feedback." arXiv:2212.08073.
- OpenAI (2023). "GPT-4 Technical Report." arXiv:2303.08774.
- OpenAI (2024). "Learning to Reason with LLMs." openai.com/research/learning-to-reason-with-llms.
- Google DeepMind (2025). "Gemini 2.5 Pro Technical Report." deepmind.google.
- Chollet, F. (2019). "On the Measure of Intelligence." arXiv:1911.01547.
- Epoch AI (2024). "FrontierMath Benchmark." epochai.org/frontiermath.
- Su, J., et al. (2021). "RoFormer: Enhanced Transformer with Rotary Position Embedding." arXiv:2104.09864.
- Turing, A. M. (1950). "Computing Machinery and Intelligence." Mind, 59(236), 433–460.
Follow new articles
Add SciHub101 to your feed reader to see new science and technology stories.
About the Author
SciHub101 is an independent science and technology publication founded and edited by Buğra Sözeri, a software engineer and app developer based in Türkiye. AI tools assist with drafting and research. Source documentation in the existing archive is uneven, and articles are being reviewed and corrected. The Editorial Team byline identifies editorial responsibility; it does not certify specialist or independent human review. Where we compare products, anything built by our own team is labelled as such — see the editorial guidelines and About page for how corrections and disclosures work.
Related Articles

How AI Is Revolutionizing Drug Discovery
AlphaFold won the 2024 Nobel. The first AI-designed drugs are in clinical trials. How AI is reshaping every stage of pharmaceutical R&D.

AI Agents Explained — How Autonomous AI Systems Actually Work
AI agents combine LLMs, tools, memory, and planning to complete multi-step tasks without human input at each step. Architecture, frameworks, real uses, and risks in 2026.

Test-Time Compute Explained — LLM Reasoning
Test-time compute lets LLMs think longer at inference. Learn how o1, o3, DeepSeek-R1, and Claude reasoning models work and when to use them.

Prompt Engineering Explained — How to Talk to AI Effectively
The same AI gives wildly different results depending on how you ask. Prompt engineering is the skill of getting LLMs to do what you want — and it's becoming one of the most valuable skills in tech.

How to Write AI Prompts That Actually Work
The quality of an AI model's output is determined largely by the quality of the input it receives. A prompt is not a search query — it is the only communication channel between the user and the model. Every word specifies or leaves ambiguous something about the desired output. This guide covers the principles behind effective prompting, the techniques with the most research evidence behind them, and the common mistakes that produce frustrating, generic results.

Complete AI Guide 2026: How Modern AI Really Works
How modern AI works: machine learning, how LLMs are trained, agents and tool use, generative models, evaluation, limits and safety, with a deep dive per topic.