Test-Time Compute Explained — LLM Reasoning

Photo: Sun God Apolo / Pexels
Test-time compute is the practice of giving a language model more computation at inference time — the moment it is actually answering you — rather than only investing compute during training. Instead of producing an answer the instant it starts generating, the model spends extra tokens working through the problem: drafting steps, checking its own logic, sometimes trying several approaches before committing to a final answer. On hard math and coding problems, this single change turned a stalled scaling story into one of the most consequential shifts of 2024–2026: OpenAI's o1 jumped from roughly 13% to about 83% on the AIME math competition simply by learning to think longer before answering, according to OpenAI's own o1 system card.
For most of the deep learning era, "scaling" was synonymous with pre-training: bigger models, more training data, more GPU-hours spent before the model ever saw a user's question. Test-time compute adds a second, independent dial. A comparatively small model, given the freedom to reason for longer, can rival a model many times its size trained the conventional way. That single insight — first demonstrated rigorously by DeepMind and UC Berkeley researchers in August 2024 — reorganized how every major AI lab designs its flagship products, and it is why your chat interface increasingly shows a "thinking" indicator before an answer appears.
The problem test-time compute solves
Every standard LLM, from the earliest GPT-3 to most 2023-era chatbots, answers with roughly the same amount of computation no matter how hard the question is. Ask it "what is 2+2" or ask it to prove a nontrivial theorem, and — architecturally — it spends about the same number of forward passes generating the response. The model has one shot, token by token, with no mechanism to pause, backtrack, or allocate extra effort to the parts that are actually difficult.
That is a strange way to solve problems. Humans do not answer a calculus exam question the same way they answer "what's your name." We slow down, write out intermediate steps, double-check arithmetic, and sometimes discard an approach halfway through and start over. A standard LLM had none of that flexibility built in — until researchers started training models specifically to use extra inference-time computation as a resource, the same way a chess engine uses extra time per move to search deeper.
The fix that stuck came from combining two prior ideas: chain-of-thought prompting (asking a model to "show its work") and reinforcement learning (rewarding the model for reasoning traces that actually lead to correct answers). Put together with enough scale, this produces what the field now calls a reasoning model: an LLM that generates a long, often hidden, internal reasoning trace before emitting its final, user-visible answer.
What counts as "test-time" versus "training-time" compute?
Training-time compute is everything that happens before the model is ever deployed: the forward and backward passes across trillions of tokens that set the model's weights. It happens once, is extremely expensive in aggregate, and is fixed once training ends. Test-time compute is spent per query, after training is finished, every single time a user asks something — you pay it repeatedly, and you can choose how much to spend based on the difficulty of the question in front of you.
How reasoning models actually generate their answers
A reasoning model does not think in the abstract sense; it thinks by generating tokens, the same way it generates a final answer, except most of those tokens are a scratchpad the user never sees by default. OpenAI, for instance, deliberately hides the raw chain-of-thought from o1 and o3 users and shows only a summarized version, partly to prevent competitors from distilling the technique by reading the raw traces, and partly because early raw traces sometimes contained unfiltered or inconsistent reasoning that OpenAI didn't want exposed directly, according to OpenAI's system card documentation.
Think of it like the difference between a student's exam answer sheet and their scratch paper. The scratch paper is where the actual thinking happens — false starts, arithmetic checks, crossed-out approaches. The exam answer is the clean, final version. A standard LLM only ever produces the exam-answer version, compressed into a single pass. A reasoning model is allowed to fill up pages of scratch paper first, and only then write the final answer, informed by everything it worked through.
Underneath that user-facing framing, labs use several concrete techniques, sometimes stacked together within a single query.
| Technique | Core idea | Where it shows up |
|---|---|---|
| Chain of Thought (CoT) | Generate step-by-step reasoning tokens before the final answer | Foundational to every reasoning model; originally described by Wei et al., 2022 |
| Self-Consistency | Sample many independent reasoning paths, take the majority answer | Research pipelines, some high-stakes production systems |
| Best-of-N sampling | Generate N full answers, use a separate verifier model to pick the best one | Reward-model-based pipelines, agentic coding tools |
| Tree of Thought (ToT) | Explore multiple reasoning branches as a search tree, prune weak ones | Research settings, some agent frameworks |
| Beam search over reasoning | Keep the top-K partial reasoning paths at each step, discard the rest | Believed to be part of the o-series internals (not publicly disclosed in detail) |
| Monte Carlo Tree Search (MCTS) | Simulate future reasoning states and back-propagate value estimates | DeepMind's AlphaProof for formal mathematics; some research reasoning systems |
| Process Reward Models (PRMs) | Score the correctness of each individual reasoning step, not just the final answer | Used during reinforcement-learning training of reasoning models |
Caption: Test-time compute is not one technique but a family of them — labs combine several within a single model or even a single query.
None of these techniques require a new base architecture. They are, largely, ways of using an existing transformer more cleverly at inference time, plus a reinforcement-learning training phase that teaches the model to generate reasoning traces that are actually useful rather than just verbose.
How is this different from simply prompting a model to "think step by step"?
Chain-of-thought prompting, the technique Wei and colleagues described in 2022, works on any capable LLM: you literally add "let's think step by step" to your prompt, and accuracy improves on multi-step problems. Reasoning models internalize that behavior through training instead of relying on the user to ask for it, and they are trained with reinforcement learning specifically to generate long, useful reasoning chains rather than short, prompted ones — often tens of thousands of tokens rather than a few hundred.
A timeline of how we got here
The shift from "prompt engineering trick" to "core product feature" happened fast, compressed into about eighteen months.
- 2022 — Jason Wei and colleagues at Google Research publish "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," showing that simply asking a model to reason step by step, with no retraining, improves performance on arithmetic and logic tasks. This is the conceptual ancestor of everything that follows.
- August 2024 — Snell, Lee, Xu, and Kumar publish "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters," giving the field its first rigorous scaling-law-style treatment of inference-time compute, and effectively naming the paradigm.
- September 12, 2024 — OpenAI launches o1-preview and o1-mini, the first widely available commercial reasoning model. o1 thinks for anywhere from seconds to minutes before answering. On the AIME 2024 math competition, OpenAI reports accuracy climbing from around 13% (GPT-4o, no extended reasoning) to roughly 83% for o1, according to OpenAI's o1 system card.
- December 20, 2024 — OpenAI announces o3, its next-generation reasoning model, and reports a score of 87.5% on ARC-AGI in high-compute mode — a benchmark specifically designed to resist memorization and prior thought to require qualitatively new architectures, not just more reasoning tokens.
- January 20, 2025 — Chinese lab DeepSeek releases DeepSeek-R1 under an MIT license, an open-weight reasoning model competitive with o1 on several public benchmarks, including 79.8% on AIME 2024 and 97.3% on MATH-500. Its accompanying paper is notable for documenting that reasoning behavior emerged largely from pure reinforcement learning, without an initial supervised fine-tuning stage, a training-methodology detail that surprised much of the research community.
- Early 2025 — DeepSeek distills R1's reasoning behavior into a family of smaller open models (R1-Distill-Qwen and R1-Distill-Llama, spanning 1.5B to 70B parameters), making locally runnable reasoning models widely available for the first time.
- February 2025 — Claude 3.7 Sonnet ships with an "extended thinking" mode that lets developers set a token budget for reasoning, exposing test-time compute as a tunable dial rather than an on/off switch.
- Through 2025 and into 2026 — Google ships Gemini reasoning variants, xAI ships Grok reasoning modes, OpenAI releases o4-mini (which, according to independent evaluations summarized by AI research outlets, becomes one of the strongest publicly benchmarked models on AIME 2024 and 2025 despite its comparatively small size), and reasoning-by-default becomes standard across frontier chat products rather than a specialized mode.
The current reasoning model landscape
Reasoning models are no longer a single lab's experiment. Every major frontier lab now ships at least one reasoning-focused variant, and the differences between them matter for anyone deciding which one to build on.
| Model family | Lab | First released | Open weights | Distinguishing trait |
|---|---|---|---|---|
| o1, o3, o4-mini | OpenAI | Sep 2024 / Dec 2024 / 2025 | No | First commercial reasoning models; o3 set the ARC-AGI benchmark; o4-mini is notably strong on AIME despite small size |
| DeepSeek-R1 | DeepSeek | Jan 2025 | Yes (MIT license) | First major open-weight reasoning model; reasoning emerged largely via pure reinforcement learning |
| Claude 3.7 / 4 (extended thinking) | Anthropic | Feb 2025 / 2025 | No | User- and developer-controllable thinking-token budget |
| Gemini 2.0 / 2.5 Flash Thinking | Google DeepMind | Dec 2024 onward | No | Optimized for low-latency, low-cost reasoning at scale |
| QwQ-32B | Alibaba | Nov 2024 | Yes | Strong open reasoning performance relative to its size |
| Grok reasoning modes | xAI | Feb 2025 onward | No | Long-context reasoning integrated with real-time data access |
Caption: By 2026, "reasoning mode" has moved from an experimental o1-only feature to a standard offering across every frontier AI lab.
After training a large reasoning model, labs frequently distill the reasoning behavior into smaller, cheaper models by fine-tuning a small model on reasoning traces generated by the large one. DeepSeek-R1-Distill-Qwen-7B, for example, inherits a substantial fraction of R1's math and coding performance while being small enough to run on a single consumer GPU — a concrete illustration that the behavior of reasoning can transfer even when the underlying parameter count drops by orders of magnitude.
Why does a smaller model with more thinking time sometimes beat a bigger model?
Because accuracy on many reasoning tasks depends more on how many candidate solution paths a model can explore and verify than on how much raw knowledge is packed into its weights. A large model still generates its answer in one pass unless it is also given reasoning capability; a smaller reasoning model that tries, checks, and revises multiple approaches can out-search a bigger model that only gets one attempt.
Real-world case studies
Numbers on a benchmark leaderboard are one thing. What test-time compute changes in practice is worth walking through concretely, because the tradeoffs differ sharply by domain.
Competitive mathematics. AIME (American Invitational Mathematics Examination) problems are the benchmark most associated with the reasoning-model era precisely because they require multi-step derivations where a single misstep invalidates the whole answer. OpenAI's reported jump from around 13% to roughly 83% between GPT-4o and o1 on AIME 2024 is not a marginal improvement — it is the difference between a model that essentially cannot do olympiad math and one that performs in the range of a strong human competitor, according to OpenAI's system card. DeepSeek-R1's 79.8% on the same benchmark, achieved by an open-weight model, showed within months that this capability was not an OpenAI-exclusive moat.
Formal theorem proving. DeepMind's AlphaProof, which does not use a standard transformer-only chain-of-thought approach but instead pairs a language model with Monte Carlo tree search over a formal proof language (Lean), reached silver-medal-equivalent performance at the 2024 International Mathematical Olympiad, according to DeepMind's own announcement. It is a useful data point that the test-time-compute idea generalizes beyond "generate more text tokens" into "search over a structured space with a verifier," which is a more general framing of the same underlying principle: spend more inference-time computation on harder problems.
Software engineering agents. Coding agents that operate over multi-file repositories — fixing a real bug, not just completing a function — lean heavily on test-time compute because a single wrong step (editing the wrong file, misunderstanding a dependency) compounds across a long agentic trajectory. Best-of-N sampling combined with a verifier that actually runs the test suite is a common production pattern precisely because it converts test-time compute into a measurable pass/fail signal, rather than relying purely on the model's own confidence.
Legal and scientific reasoning. A March 2025 evaluation on arXiv comparing OpenAI's o1, DeepSeek-R1, and related reasoning models specifically on legal reasoning tasks found that test-time-scaled models offered meaningful improvements over non-reasoning baselines on multi-step legal analysis, though the paper also documents cases of "underthinking," where reasoning models abandon a promising line of reasoning too early and switch approaches, costing both accuracy and tokens (Evaluating Test-Time Scaling LLMs for Legal Reasoning, arXiv, March 2025). This kind of failure mode — reasoning that looks thorough but is actually scattered — is one reason test-time compute is not a universal accuracy dial; more thinking tokens do not guarantee better thinking.
When test-time compute is worth the cost — and when it isn't
Reasoning models cost more, respond slower, and are not a strict upgrade over standard LLMs. The decision of when to use one is really a decision about the cost of being wrong versus the cost of waiting.
Good fit:
- Math problems, especially competition-style or multi-step derivations (AIME, olympiad-level proofs)
- Competitive programming, debugging, and any coding task with a verifiable test suite
- Scientific reasoning: proofs, multi-step derivations, experimental design questions
- Multi-step planning and agentic tool use, where an early mistake compounds
- Anything where an incorrect answer is expensive to discover later (financial modeling, medical literature synthesis, legal analysis)
Bad fit:
- Simple factual lookups ("what is the capital of France?") where a standard model is already at or near ceiling accuracy
- Creative writing and brainstorming, where speed and volume of ideas matter more than verified correctness
- Real-time chat and voice assistants, where a multi-second "thinking" delay breaks the interaction
- High-volume, low-margin inference, since reasoning-token cost scales roughly linearly with how long the model thinks
A single reasoning query, particularly one run at a high-compute setting like OpenAI's o3 in its ARC-AGI configuration, can cost tens to thousands of times more than a standard query in both latency and dollars — reports on o3's high-compute ARC-AGI runs put per-task inference cost in the thousands of dollars. Because of that spread, most production systems now route: a fast classifier or heuristic sends easy questions to a cheap, non-reasoning model and only escalates genuinely hard questions to a reasoning model. Anthropic's "extended thinking" and similar developer-facing token budgets exist precisely so engineering teams can set that threshold themselves rather than paying reasoning-model prices for every query by default.
How much more does a reasoning query cost compared to a standard one?
There is no single fixed multiplier — it depends on the difficulty of the question and how the lab prices reasoning tokens — but reasoning models frequently generate anywhere from a few hundred to tens of thousands of hidden reasoning tokens before the visible answer, and every one of those tokens is billed and takes wall-clock time, unlike a standard model's single-pass response.
Why this matters beyond any single benchmark
Test-time compute functions as a second, largely independent scaling axis for AI capability. Pre-training scaling laws — the Kaplan et al. results from 2020 and the Chinchilla-optimal training results from Hoffmann et al. in 2022 — established a fairly predictable relationship: more parameters, more training data, and more training compute produce better models, in a roughly quantifiable way. What those laws did not address is what happens once you fix a trained model and instead let it spend more computation per answer. The 2024–2025 reasoning paradigm answers that question, and the Snell et al. result — that inference-time compute scaling can, on the right tasks, be more compute-efficient than pre-training scaling — gives labs a second lever to pull when one starts showing diminishing returns.
That matters directly for the "is scaling hitting a wall" debate that has run through the AI industry since 2023. If pre-training gains from simply adding more parameters and data begin to flatten for a given architecture, test-time compute offers an alternative path to capability gains that does not require training an entirely new, larger foundation model from scratch. It is not a replacement for pre-training — a weak base model given more thinking time does not become a strong model — but it decouples "how capable is this system" from "how large is this system," which changes both the technical and business calculus of building frontier AI.
It also reshapes the hardware and economics conversation. Inference compute, once treated mostly as a fixed operating cost to be minimized, is now a strategic capability in its own right, because a lab that can deliver more useful reasoning tokens per dollar and per second has a direct product advantage. That is part of why NVIDIA, AMD, Cerebras, and Groq have all pushed inference-optimized hardware and software stacks specifically targeting low-latency, high-throughput generation of long reasoning chains, rather than only optimizing for training throughput as in prior hardware generations.
What people get wrong about reasoning models
"The model is literally thinking like a human." It is generating tokens according to a learned reasoning policy, optimized during reinforcement-learning training to correlate with correct final answers. That is a genuinely different, more structured computational process than a single-pass answer, but it is not evidence of the kind of general, embodied reasoning humans do — it is a specific, trained skill for producing useful token sequences before answering.
"The visible chain-of-thought is what the model is actually computing." Labs including OpenAI often show a summarized or filtered version of the reasoning trace, not the raw internal tokens, and independent research has documented cases where a model's stated reasoning does not fully match the computation that produced its answer — a property researchers call "unfaithful" reasoning. Treat a displayed chain-of-thought as a helpful but not fully reliable explanation, not a verified audit trail.
"More test-time compute always means a better answer." The arXiv legal-reasoning evaluation from March 2025 documents "underthinking" failures where models abandon a good line of reasoning prematurely, and independent work such as "It's Not That Simple: An Analysis of Simple Test-Time Scaling" (2025) shows that naive scaling of reasoning length does not translate linearly into accuracy gains — returns diminish, and past a point, more tokens can even correlate with worse answers on some tasks due to reasoning drift.
"Test-time compute replaces the need for a good base model." It does not. A reasoning model still relies on the underlying pre-trained network's knowledge and pattern-recognition ability; test-time compute amplifies existing capability through search and self-correction, it does not manufacture capability that was never there in the base model. Distillation experiments make this visible: distilling R1's reasoning traces into a weak base model produces smaller gains than distilling into a strong one, because the base model's ceiling still bounds what extra thinking time can achieve.
"Reasoning models are strictly an upgrade, so you should always use one." For latency-sensitive, low-stakes, or high-volume use cases, a standard model at a fraction of the cost and response time will often perform just as well in practice, because the extra reasoning capacity is wasted on questions that were never hard in the first place.
The energy and infrastructure question
One consequence of shifting compute from training-time to test-time that gets less attention than benchmark scores is energy use. A September 2025 analysis on arXiv, "Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling," examines how test-time scaling changes the energy profile of AI systems: because reasoning tokens are generated and billed per query, and every deployed query pays that cost repeatedly, the aggregate energy footprint of inference compute grows with usage in a way that one-time training compute does not. That paper frames test-time scaling as introducing a genuinely new axis of energy cost that infrastructure planning has to account for, distinct from the well-studied energy cost of training runs. For a company deploying reasoning models at scale, that is a direct argument in favor of routing — sending only the fraction of traffic that actually needs deep reasoning to the expensive reasoning path, and keeping the rest on cheaper, faster models.
Frequently Asked Questions
Q: Is test-time compute the same thing as chain-of-thought prompting? A: No. Chain-of-thought prompting, from Wei et al.'s 2022 paper, is a technique you can apply to any capable LLM by literally asking it to reason step by step. Test-time compute is the broader paradigm of spending extra inference-time computation, which includes chain-of-thought as one method alongside self-consistency voting, best-of-N sampling, tree search, and verifier-guided generation, often combined with reinforcement-learning training so the behavior happens by default rather than requiring a special prompt.
Q: When did OpenAI release the first commercial reasoning model? A: OpenAI launched o1-preview and o1-mini on September 12, 2024. It was the first widely available model specifically trained to generate long internal reasoning traces before answering, and OpenAI reported its AIME 2024 accuracy jumping from roughly 13% to about 83% as a result.
Q: What is the difference between o1 and o3? A: o3, announced December 20, 2024, is OpenAI's successor to o1 with stronger reasoning performance. Its most notable public result was 87.5% on ARC-AGI in a high-compute configuration, a benchmark specifically designed to resist memorization that its creators previously believed would require new architectural approaches rather than more test-time reasoning.
Q: Can I run a reasoning model locally on my own hardware? A: Yes. DeepSeek-R1 was released with open weights under an MIT license, and its distilled variants (ranging roughly from 1.5B to 70B parameters) can run on consumer GPUs, with smaller versions running on a single card. Alibaba's QwQ-32B is another strong openly available reasoning model.
Q: Does test-time compute replace the need for pre-training scaling? A: No — the two compose rather than substitute for each other. A model still needs a capable base built through pre-training; test-time compute amplifies what is already present in that base model through search and self-correction, it does not create capabilities that were never learned during training.
Q: Why is running a reasoning model so much more expensive than a standard model? A: A reasoning model can generate anywhere from a few hundred to tens of thousands of hidden reasoning tokens before producing its visible answer, and every one of those tokens is computed and billed. High-compute configurations, like the one OpenAI used for o3's ARC-AGI results, have reportedly cost thousands of dollars per task in inference spend.
Q: Does a reasoning model's visible "thinking" text accurately reflect what it actually computed? A: Not necessarily. Some labs show a summarized or filtered version of the reasoning trace rather than the raw tokens, and research has documented cases of "unfaithful" reasoning, where a model's stated chain-of-thought does not fully match the computation behind its final answer. Treat displayed reasoning as informative but not a guaranteed audit trail.
Q: What is DeepSeek-R1 and why did it matter so much when it was released? A: DeepSeek-R1 is an open-weight reasoning model released by the Chinese lab DeepSeek on January 20, 2025, under an MIT license. It matched OpenAI's o1 on several public benchmarks, scoring 79.8% on AIME 2024 and 97.3% on MATH-500, and its accompanying paper documented that much of its reasoning behavior emerged from reinforcement learning alone, without an initial supervised fine-tuning stage — a training detail that reshaped how other labs approached reasoning-model training.
Q: Can more test-time compute ever make answers worse? A: Yes, in specific documented cases. Research on "underthinking" shows reasoning models sometimes abandon a promising approach prematurely and switch strategies, wasting tokens and hurting accuracy, and other analyses have found that naively scaling reasoning length does not translate linearly into better accuracy — gains diminish and can, on some tasks, reverse.
Q: How do companies decide when to send a query to a reasoning model versus a standard model? A: Most production systems use a routing layer: a fast classifier, heuristic, or the user's own explicit setting determines whether a query is simple enough for a cheap, standard model or difficult enough to justify the added latency and cost of a reasoning model. Anthropic's "extended thinking" mode, for example, lets developers set a token budget for reasoning explicitly rather than paying reasoning-model prices by default for every request.
The takeaway
Test-time compute proved that model capability is not fixed the moment training ends. By letting a model spend variable computation per question, rather than the same fixed amount regardless of difficulty, labs unlocked a second scaling axis that, on the right tasks, can substitute for a much larger pre-trained model. The catch is that this extra capability is not free or universal: it costs real money and real latency per query, it does not help on tasks that were never reasoning-bottlenecked, and the visible "thinking" a model shows you is a helpful but imperfect window into what it actually computed. The practical skill for anyone building on these models in 2026 is not "always use the reasoning model" — it is knowing which questions are worth the wait.
Read next:
Follow new articles
Add SciHub101 to your feed reader to see new science and technology stories.
About the Author
SciHub101 is an independent science and technology publication founded and edited by Buğra Sözeri, a software engineer and app developer based in Türkiye. AI tools assist with drafting and research. Source documentation in the existing archive is uneven, and articles are being reviewed and corrected. The Editorial Team byline identifies editorial responsibility; it does not certify specialist or independent human review. Where we compare products, anything built by our own team is labelled as such — see the editorial guidelines and About page for how corrections and disclosures work.
Related Articles

Context Window Explained — LLM Memory Limits
Context windows define how much text an LLM can process at once. Learn token limits, the lost-in-the-middle problem, and 2026 model comparisons.

RLHF vs RLAIF vs DPO — Alignment Explained
How LLMs learn to be helpful: RLHF, RLAIF, DPO, and Constitutional AI compared. Methods used by ChatGPT, Claude, Llama, and open models.

How Large Language Models Actually Reason — Inside ChatGPT, Claude, and Gemini (2026)
Do LLMs actually reason or just predict text? A deep technical guide to transformer attention, chain-of-thought, test-time compute, and frontier model benchmarks in 2026.

AI Reasoning and Chain of Thought: How Models Learn to Think Step by Step
Chain-of-thought prompting and reasoning models have changed what language models can solve. Here is the evidence on why writing out steps helps, how it is trained now, and where the steps stop being trustworthy.

How AI Models Are Trained: The Full Pipeline Explained
Training a large language model is not a single act — it is a five-stage industrial process that begins with petabytes of scraped text, passes through weeks of GPU computation consuming millions of dollars of electricity, and ends with a model that can hold a coherent conversation. Each stage has a specific job, a distinct failure mode, and a measurable cost. Here is how the whole pipeline works, from raw data to deployed product.

How to Use Claude AI: Anthropic's Chatbot Explained, Features, Strengths, and Limitations
Claude is an AI chatbot built by Anthropic and available on Claude.ai, through the API, and via third-party integrations. It excels at long-form writing, coding, analysis of complex documents, and reasoning through multi-step problems. Unlike ChatGPT, Claude does not train on conversations, offers higher context limits, and produces more thorough outputs. This guide explains what Claude is, how to access it, what it does better than competitors, its actual limitations, and how to prompt it effectively.