AI Training vs. Inference: What Actually Happens When AI Learns and Thinks

When people talk about AI "learning" and AI "thinking," they are usually mixing up two processes that are computationally, economically and architecturally different. Training is when a model acquires its capabilities: its internal numbers, called weights or parameters, are adjusted again and again until it performs well. Inference is when a trained model applies those capabilities to new input, with its weights left untouched. Confusing the two leads to persistent misunderstandings about how AI works, what it costs, and what it can and cannot do.

The distinction matters practically. It explains why a chatbot does not remember yesterday's conversation unless a product feature stores it, why a model's knowledge stops at a cutoff date, why training a frontier model costs tens of millions of dollars while a single answer costs a fraction of a cent, and why the hardware used for each job is tuned differently.

Key takeaways

• Training adjusts a model's parameters to reduce prediction error over a dataset; inference applies frozen parameters to new inputs.
• A common rule of thumb from the scaling-law literature: training a transformer costs about 6 floating-point operations (FLOPs) per parameter per training token; a forward pass at inference costs about 2 FLOPs per parameter per token.
• Training needs far more memory than inference: with mixed-precision Adam, model states alone take about 16 bytes per parameter, versus about 2 bytes per parameter for 16-bit inference weights.
• A model does not update its weights during ordinary inference. Your conversations do not change it.
• Training is a large, mostly one-off cost; inference is a recurring cost paid on every request, and it grows with usage.

The core concept

Training is the process of adjusting a neural network's parameters to make it good at a task. For a large language model (LLM), pre-training means showing the model enormous quantities of text and, step after step, nudging billions of numerical weights so that it better predicts the next token (a word or word-piece). Our explainer How AI Models Are Trained walks through the whole pipeline, from data collection to alignment.

Inference is the process of running a trained model to produce an output for a new input. The weights are frozen. Given a prompt, the model performs a fixed sequence of mathematical operations (mostly matrix multiplications and attention calculations) and produces output tokens one at a time.

An analogy: training is like years of study; inference is like answering an exam question using what you learned. A doctor answering a question is not re-learning medicine in that moment. The analogy breaks down in one important way, though. People learn continuously from every experience. A deployed language model does not; any "learning" after deployment requires a separate, deliberate training step.

How training works, step by step

A training run processes the dataset (or very large portions of it) in batches. At each step:

1. Forward pass. A batch of examples is fed through the model, which produces predictions, for an LLM a probability for every possible next token.
2. Loss. The predictions are compared with the actual next tokens, and a single number, the loss, measures how wrong the model was. Cross-entropy loss is standard for language models.
3. Backward pass (backpropagation). Using the chain rule of calculus, the gradient of the loss with respect to every parameter is calculated. The gradient says, for each weight, which direction and roughly how much to move it to reduce the loss.
4. Optimizer update. An optimizer, most commonly Adam or a variant such as AdamW, uses those gradients to update every weight. Adam keeps two extra running statistics per parameter (an average of past gradients and of their squares), which is one reason training needs so much memory.
5. Repeat. This loop runs for hundreds of thousands to millions of steps.

The mathematics is simple to state and enormously expensive to perform. Every step touches every parameter twice (forward and backward), and the backward pass is roughly twice as costly as the forward pass.

The 6ND rule of thumb

In their 2020 paper Scaling Laws for Neural Language Models, Kaplan and colleagues at OpenAI estimated training compute for a transformer as C ≈ 6N FLOPs per training token, where N is the number of (non-embedding) parameters, and the forward pass alone as roughly 2N FLOPs per token (arXiv:2001.08361). Multiply by the number of training tokens D, and total training compute is approximately 6 × N × D.

The rule works remarkably well. Meta's Llama 3 paper reports that its flagship 405-billion-parameter model was pre-trained on 15.6 trillion text tokens using 3.8 × 10²⁵ FLOPs (Llama 3 paper, arXiv:2407.21783). Plug those numbers into the rule: 6 × 405 × 10⁹ × 15.6 × 10¹² ≈ 3.79 × 10²⁵. The back-of-envelope estimate lands within 1% of the reported figure.

How big is "big"? Real training runs

Model	Parameters	Training tokens	Reported compute / hardware	Source
Chinchilla (DeepMind, 2022)	70B	1.4T	Same compute budget as the 280B Gopher model	Hoffmann et al., arXiv:2203.15556
Llama 2 family (Meta, 2023)	7B–70B	2T	3.3M A100-80GB GPU-hours for pre-training across the family	Touvron et al., arXiv:2307.09288
Llama 3 405B (Meta, 2024)	405B	15.6T	3.8 × 10²⁵ FLOPs on up to 16,000 H100 GPUs	Llama 3 paper, arXiv:2407.21783

The Chinchilla paper is also where the much-quoted guideline of roughly 20 training tokens per parameter for compute-optimal training comes from: 1.4 trillion tokens divided by 70 billion parameters is exactly 20. Later models, including Llama 3 (about 38 tokens per parameter for the 405B model), deliberately train on more data than that, because a smaller model trained longer is cheaper to run at inference time. That trade-off is a direct link between training decisions and inference economics.

What training costs in money

Training cost estimates are uncertain because companies rarely publish them. Stanford's AI Index Report 2024, working with the research group Epoch AI, estimated the compute cost of training GPT-4 at about $78 million and Google's Gemini Ultra at about $191 million. These are estimates of compute only, and other methods give different numbers, so treat them as rough orders of magnitude. Epoch AI's own analysis of 45 frontier models found that training costs have grown about 2–3 times per year since 2016 and that, if the trend continues, the largest runs would exceed $1 billion by 2027. It also found that hardware makes up 47–67% of total development cost, R&D staff 29–49%, and energy only 2–6% (Epoch AI).

How inference works, step by step

For a deployed language model answering a prompt:

1. Tokenize the input text into token IDs.
2. Embed each token as a vector.
3. Run the layers. The vectors pass through the transformer's layers in sequence; each layer applies attention (letting each token look at the others) and a feed-forward network. For a visual explanation of attention, see How Transformers Pay Attention to Words.
4. Score the vocabulary. The final layer produces a score (logit) for every token in the vocabulary.
5. Sample the next token from those scores, using settings such as temperature.
6. Append and repeat. The new token is added to the sequence and steps 3–6 repeat until the answer is complete.

Inference has two distinct phases with different performance characteristics:

• Prefill: the whole prompt is processed at once. This is compute-heavy and highly parallel, because all prompt tokens are known.
• Decode: output tokens are produced one at a time, each depending on the one before. Each step must read all of the model's weights from memory to produce a single token per sequence, so decoding tends to be limited by memory bandwidth rather than raw arithmetic. Batching many users' requests together is how serving systems use the hardware efficiently.

The KV cache

Generating token 500 requires attention over tokens 1 to 499. Recomputing everything from scratch each time would be wasteful, so inference engines store each layer's attention keys and values for every past token in a key-value (KV) cache.

The cache is large. The authors of the vLLM paper give a worked example: for the 13-billion-parameter OPT model, the KV cache for a single token takes about 800 KB, calculated as 2 (key and value) × 5,120 (hidden size) × 40 (layers) × 2 bytes. A 2,000-token conversation therefore needs about 1.6 GB of cache for one user. On an NVIDIA A100 with 40 GB of memory serving that 13B model, they measured roughly 65% of memory taken by the weights and close to 30% by the KV cache. They also found that in earlier serving systems only 20.4–38.2% of KV cache memory actually held token states, the rest being lost to fragmentation and over-reservation; their PagedAttention technique raised throughput by 2–4× at the same latency (Kwon et al., arXiv:2309.06180). Our article What Is vLLM? covers that engine in more detail.

The KV cache is also why long context windows are expensive: cache size grows linearly with the number of tokens in context. See Context Window Explained for the user-facing side of this limit, and Prompt Caching Explained for how providers reuse cached prompt prefixes to cut cost.

The memory gap: why training needs so much more hardware

The crucial technical difference is the gradient. Training must compute gradients, which means storing the intermediate results (activations) from the forward pass so the backward pass can use them, plus the gradients themselves, plus the optimizer's statistics.

The ZeRO paper from Microsoft researchers quantified the model-state part precisely. With mixed-precision training and the Adam optimizer, a model with Ψ parameters needs 2Ψ bytes for 16-bit weights, 2Ψ bytes for 16-bit gradients, and 12Ψ bytes for optimizer states (a 32-bit copy of the weights plus Adam's two statistics), a total of 16Ψ bytes, before counting activations (Rajbhandari et al., arXiv:1910.02054).

Inference with 16-bit weights needs about 2 bytes per parameter for the weights, plus the KV cache.

	Training (mixed-precision Adam)	Inference (16-bit weights)
Weights	2 bytes/param	2 bytes/param
Gradients	2 bytes/param	none
Optimizer states	12 bytes/param	none
Activations	Stored for backward pass; large, grows with batch and sequence length	Only transient, plus KV cache
Model states for a 70B model	~1,120 GB (16 × 70B)	~140 GB (2 × 70B)
FLOPs per token	~6N	~2N

Sources: memory per parameter from ZeRO (arXiv:1910.02054); FLOPs per token from Kaplan et al. (arXiv:2001.08361). The 70B figures are our arithmetic.

For a 70B model, the training-side model states alone are about eight times the inference weights, and they cannot fit on any single GPU. This is why training is spread across thousands of accelerators using data parallelism, tensor parallelism and pipeline parallelism, while inference for the same model can run on one server with several GPUs.

Quantization widens the gap further

At inference, weights can be stored at lower precision, 8-bit or even 4-bit integers instead of 16-bit floats. A 70B model at 4 bits needs roughly 35 GB for its weights, which is the difference between needing a multi-GPU server and fitting on a single high-memory card. Quantization usually costs a little accuracy, and how much depends on the method and the model. It is primarily an inference optimization, because training needs higher precision to accumulate many tiny gradient updates reliably. Our guide to LLM quantization explains INT8 and INT4 methods, and The Hidden Cost of Running a Local LLM on Your Laptop shows what the trade-offs look like on consumer hardware.

A worked comparison: one training run vs. a year of answers

The FLOP rules let us compare training and inference on the same scale. Take the Llama 3 405B model:

• Training: about 6 × N × D = 3.8 × 10²⁵ FLOPs (reported).
• Inference: about 2 × N = 2 × 405 × 10⁹ ≈ 8.1 × 10¹¹ FLOPs per generated token (ignoring attention over long contexts).

Divide one by the other: the pre-training compute equals the compute needed to generate about 3 × D, or roughly 47 trillion tokens of output. That sounds like a lot, and for a single company's internal tool it would take forever to reach. But a popular service answering hundreds of millions of requests a day, each with hundreds or thousands of tokens of input and output, can process trillions of tokens a month. At that scale, cumulative inference compute catches up with and then passes the one-time training compute.

This is the economic reason the industry obsesses over inference efficiency. A 10% saving on training is paid once; a 10% saving on inference is paid back on every request for the life of the model. It also explains why labs sometimes train smaller models on more data than the Chinchilla-optimal ratio: extra training compute buys a model that is cheaper to serve forever after.

Training vs. inference side by side

	Training	Inference
Goal	Find good weights	Use the weights to produce outputs
Weights	Updated every step	Frozen
Passes	Forward + backward	Forward only
Compute per token	~6N FLOPs	~2N FLOPs
Memory	Weights, gradients, optimizer states, activations	Weights + KV cache
Bottleneck	Total compute and inter-GPU communication	Memory bandwidth (decode), latency, batching
Frequency	Once per model version, plus fine-tunes	Every request, continuously
Typical scale	Thousands of accelerators for weeks or months	One GPU to one server per model replica, many replicas
Cost pattern	Large upfront cost	Recurring cost proportional to usage
Key metrics	Loss, throughput (tokens/sec trained), utilization	Time to first token, tokens/sec, cost per million tokens

Fine-tuning, distillation and RAG: the in-between zone

Fine-tuning takes a pre-trained model and continues training on a smaller, targeted dataset, such as instruction-following examples, a company's support transcripts, or code. It modifies weights, so technically it is training, but it uses a tiny fraction of pre-training compute. Parameter-efficient methods such as LoRA update only small added matrices instead of all weights, cutting memory needs further.

Post-training is the umbrella term for everything after pre-training: supervised fine-tuning, reinforcement learning from human feedback (RLHF), and preference methods. It shapes how a model behaves, how it follows instructions and what it refuses, rather than what it fundamentally knows.

Distillation trains a small "student" model to imitate a large "teacher". The teacher runs inference to generate outputs, and the student trains on them. It is training paid for with inference, a neat example of the two phases feeding each other. See AI Model Distillation Explained.

Retrieval-augmented generation (RAG) changes what a model knows at inference time without touching its weights: relevant documents are retrieved from a database and placed in the prompt. It is the most common practical answer to the knowledge-cutoff problem. See What Is RAG?

Real-world implications

Models don't learn from your conversations in real time. When you chat with an assistant, your messages do not change the model's weights. Within a conversation, earlier messages are kept in the context window and re-read at each turn; across conversations, nothing persists unless the product explicitly stores information (for example a "memory" feature) and re-inserts it into later prompts. Whether a provider uses conversations to train future models is a separate policy question, governed by its data terms, and would happen in a later training run, not during your chat. Hosted providers such as Together AI and Fireworks AI sell exactly this serving capacity, and models like Grok are trained once and then served to users through inference.

Same prompt, different answers. Because the next token is sampled from a probability distribution, the same frozen model can give different answers to the same prompt. That variation comes from sampling at inference time, not from the model changing.

Latency vs. throughput. For interactive chat, the time to first token and smooth streaming matter most. For batch work, such as summarizing a million documents overnight, total tokens per second per dollar matters most. Serving systems trade these off through batch size: larger batches raise throughput but can increase each user's wait.

Edge and on-device inference. Phones and laptops can run small models locally thanks to quantization, distillation and efficient architectures. Training, even fine-tuning, of anything but small models remains a data-center job.

Energy and carbon. Meta's Llama 2 paper reported that pre-training its model family used 3.3 million A100 GPU-hours and estimated total emissions of 539 tonnes of CO₂-equivalent, which Meta stated were fully offset (arXiv:2307.09288). Epoch AI's analysis puts energy at only 2–6% of the total development cost of frontier models. The full climate effect of inference is harder to measure, because it is spread across many data centers and depends on usage volume and on the carbon intensity of each grid.

Common misconceptions

"The model is thinking while the typing indicator shows." What you see is the model generating tokens and streaming them as they are produced. Each token takes one forward pass through the network. Reasoning models do produce extra internal tokens before answering, which is a form of extra inference computation, not training.

"A model gets smarter the more you use it." Not in standard deployment. Any personalization you notice usually comes from stored preferences or memory being inserted into the prompt.

"Training and inference use different models." Inference runs exactly the weights that training produced, sometimes converted to a lower precision. They are two phases in the life of one model.

"You need a supercomputer to run inference." For the largest models you need serious server hardware, but small open models run on consumer GPUs and laptops, especially when quantized.

"More training always makes a model better." Training too long on the same data can lead to overfitting, where the model memorizes training examples and generalizes worse. Scaling-law research shows that the right balance between model size and data size matters more than simply training longer.

The future: where the boundary is blurring

Test-time compute. Reasoning models improve answers on hard problems by spending more computation at inference, generating long chains of intermediate reasoning, or sampling several answers and choosing the best. This makes inference compute a lever for capability, not just a running cost. See Test-Time Compute Explained.

Continual learning. Standard LLMs have a fixed knowledge cutoff. Research on letting models learn after deployment without "catastrophic forgetting" (losing old skills when learning new ones) is active, but there is no widely deployed solution for frontier models.

In-context learning. Models can pick up a new pattern from a few examples in the prompt, without any weight update. It looks like learning but happens entirely during inference, and disappears when the context is gone.

Federated learning. Training on data that stays on users' devices or in separate organizations, sending only model updates to a central server, is used for some on-device models and remains an active research area for larger ones.

Specialized hardware. Because inference is limited by memory bandwidth during decoding, chips and serving software designed specifically for inference have become their own category, separate from the large clusters optimized for training.

FAQ

Q: What is the difference between AI training and inference in one sentence? A: Training is the expensive process that sets a model's weights by repeatedly correcting its predictions with gradients; inference is running the finished model, with its weights frozen, to produce outputs for new inputs.

Q: How much compute does training take compared with inference? A: A widely used rule from Kaplan et al. (2020) is about 6 FLOPs per parameter per training token for training and about 2 FLOPs per parameter per token for a forward pass. Llama 3 405B's reported training compute of 3.8 × 10²⁵ FLOPs matches 6 × 405 billion × 15.6 trillion almost exactly.

Q: Why does training need so much more memory than inference? A: Training stores gradients, optimizer states and forward-pass activations in addition to the weights. With mixed-precision Adam, model states take about 16 bytes per parameter, according to the ZeRO paper, compared with about 2 bytes per parameter for 16-bit inference weights.

Q: Does ChatGPT or any chatbot learn from my conversation while I talk to it? A: No. The weights are frozen during inference. The model sees earlier messages because they are kept in the context window, and products with memory features store notes and add them back to the prompt. Any learning would happen in a separate, later training run.

Q: What is pre-training vs. post-training? A: Pre-training is large-scale next-token prediction on raw text, which gives the model its broad knowledge and language ability. Post-training includes supervised fine-tuning, RLHF and preference methods, which shape how the model follows instructions and behaves.

Q: How much does it cost to train a frontier model? A: Companies rarely disclose it. Stanford's AI Index 2024 estimated about $78 million in compute for GPT-4 and about $191 million for Gemini Ultra, and Epoch AI finds frontier training costs have grown roughly 2–3 times per year since 2016.

Q: What is the difference between fine-tuning and prompt engineering? A: Prompt engineering changes behavior only through the input text and leaves the weights untouched, so it is cheap and reversible. Fine-tuning changes the weights through additional training, which suits persistent changes in style, format or specialized vocabulary.

---

Read next:

• How Large Language Models Work: Transformers, Attention, and Why They Hallucinate
• How AI Models Are Trained: The Full Pipeline Explained
• How Does ChatGPT Actually Work? A Plain-Language Guide to LLMs

Sources

• Kaplan, J. et al. (2020), Scaling Laws for Neural Language Models, arXiv:2001.08361.
• Hoffmann, J. et al. (2022), Training Compute-Optimal Large Language Models (Chinchilla), arXiv:2203.15556.
• Rajbhandari, S. et al. (2019), ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, arXiv:1910.02054.
• Touvron, H. et al. (2023), Llama 2: Open Foundation and Fine-Tuned Chat Models, arXiv:2307.09288.
• Llama Team, Meta (2024), The Llama 3 Herd of Models, arXiv:2407.21783.
• Kwon, W. et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM), arXiv:2309.06180.
• Epoch AI, How Much Does It Cost to Train Frontier AI Models?