Why AI Needs So Much Computing Power

Training GPT-3, a model with 175 billion parameters, took about 3.14 × 10²³ floating-point operations, according to the table of training compute in its 2020 paper (Brown et al., 2020). That is a 314 followed by 21 zeros. Spread across one graphics processor running at a realistic sustained speed, the job would take centuries. The reason is one operation, matrix multiplication, repeated at enormous scale. This article explains why that operation dominates AI's compute bill, why GPUs rather than CPUs do the work, and what the scaling-law research says about where all that compute goes.

What a neural network is actually computing

Matrix multiplication is a mathematical operation that combines two grids of numbers into a third grid. Each output value comes from multiplying pairs of numbers across a row and down a column, then adding the results. It sounds mundane. It is also, almost in its entirety, what a neural network does when it processes information.

A large language model's forward pass, the step that turns an input into an output, is a long chain of these multiplications. Input text is converted into numerical vectors, those vectors are multiplied against a matrix of learned weights, the result passes through a nonlinear function that lets the network model curves rather than straight lines, and the output is multiplied against another weight matrix. This repeats across every layer. GPT-3 has 96 layers, and each layer performs several distinct matrix multiplications on every token it processes (Brown et al., 2020).

Multiply that per-token cost by a training set of hundreds of billions of tokens and the total number of multiply-and-add operations climbs past what any human intuition can hold. GPT-3 trained on 300 billion tokens. This is not a design inefficiency that better engineering will remove. It is the computation itself. A network's "knowledge" is stored entirely as the numbers inside those weight matrices, and there is no shortcut for computing the output of a matrix multiplication without doing the multiplication.

A useful analogy is a spreadsheet. Imagine a spreadsheet with 175 billion cells, each holding a number, where producing one word of output requires reading and combining all of them, and where learning requires adjusting every cell a tiny amount after each batch of text. The cost does not come from any single clever calculation. It comes from doing something simple a staggering number of times.

Epoch AI estimates that the training compute of notable AI models grew about 4.1 times per year (90 percent confidence interval 3.7 to 4.6) from 2010 to May 2024, and that frontier models grew faster, about 5.3 times per year over the same span. Since about 2018, Epoch puts frontier growth at about 4.2 times per year (Epoch AI, "Training Compute of Frontier AI Models Grows by 4-5x per Year," 2024).

How to count the cost: a back-of-envelope calculation

Researchers use a simple rule to estimate training compute. The DeepMind team behind the Chinchilla paper writes it as FLOPs ≈ 6ND, where N is the number of parameters and D is the number of training tokens (Hoffmann et al., 2022). A FLOP is one floating-point operation, such as one multiplication or addition of decimal numbers. (Written FLOPs, the plural, it means a count of operations; written FLOP/s, it means operations per second.)

The factor of six has a plain explanation. A forward pass costs about two operations per parameter per token (one multiply, one add). Backpropagation, the process of working out how each weight should change, costs roughly twice as much as the forward pass. Two plus four makes six.

Test the rule on a known case. GPT-3 has 175 billion parameters and trained on 300 billion tokens (Brown et al., 2020). Six times 175 billion times 300 billion is about 3.15 × 10²³, almost exactly the 3.14 × 10²³ the paper reports. The rule is a rough approximation, but on this case it lands within a fraction of a percent.

How long would that take on one chip?

Here is an illustration of the scale. The GPT-3 paper says the models were trained on V100 GPUs on a cluster provided by Microsoft (Brown et al., 2020). Assume a single GPU sustains 28 teraFLOP/s, a figure well below any chip's peak rating because real workloads never hit peak. Then 3.14 × 10²³ operations divided by 2.8 × 10¹³ operations per second is about 1.1 × 10¹⁰ seconds, or roughly 355 years. The 28 teraFLOP/s is an assumption for this illustration, not a measured value, but changing it by a factor of two still leaves a result measured in centuries.

That is why training uses thousands of chips working together for weeks or months. Doing it faster needs more parallelism, and more parallelism needs more hardware and more electricity.

How far has the rule scaled?

Model	Year	Parameters	Training tokens	Tokens per parameter	Approximate training compute
GPT-3	2020	175 billion	300 billion	about 1.7	3.14 × 10²³ FLOP
Gopher	2021	280 billion	300 billion	about 1.1	5.76 × 10²³ FLOP
Chinchilla	2022	70 billion	1.4 trillion	20	same budget as Gopher

Sources: Brown et al., 2020; Hoffmann et al., 2022. Tokens per parameter is computed from the two preceding columns. The Chinchilla paper states that Gopher's budget was 5.76 × 10²³ FLOPs and that Chinchilla used the same budget; Appendix F of the paper gives a more careful accounting of about 6.3 × 10²³.

Why the parameter count alone doesn't tell the whole story

Parameters are the individual learned numbers inside a model's weight matrices, and headlines often use parameter count as shorthand for size and cost. But training cost depends on parameters multiplied by tokens processed, not parameters alone. A smaller model trained longer can cost as much compute as a larger model trained briefly.

Take two hypothetical models. A 7-billion-parameter model trained on 2 trillion tokens uses 6 × 7 × 10⁹ × 2 × 10¹² ≈ 8.4 × 10²² FLOPs. A 30-billion-parameter model trained on 300 billion tokens uses 6 × 30 × 10⁹ × 3 × 10¹¹ ≈ 5.4 × 10²² FLOPs. The smaller model costs more to train, even though the second one sounds bigger by the number quoted in headlines.

That is why compute, not parameter count, is the figure labs budget against. A model's parameter count measures how much it could learn. Compute measures how much work went into teaching it.

Scaling laws: what Kaplan and Chinchilla actually found

In 2020, Kaplan and colleagues showed that a language model's cross-entropy loss (a measure of how badly it predicts the next token) follows power-law trends with model size, dataset size, and training compute, spanning more than seven orders of magnitude. They also found that changes to network shape, such as width or depth, mattered little across a broad range, and that larger models were much more sample-efficient (Kaplan et al., 2020). The practical consequence is that a lab can predict, before spending the money, roughly how much better a bigger run will be. That predictability is what makes very expensive training runs easier to justify before they finish.

Kaplan's work also suggested that the most compute-efficient training used very large models on a modest amount of data and stopped well before convergence (Kaplan et al., 2020). Our article on scaling laws covers that history in more detail.

In 2022, a DeepMind team led by Jordan Hoffmann revisited the question. They trained more than 400 language models, ranging from about 70 million to over 16 billion parameters, on 5 to 500 billion tokens, and reached a different conclusion: for compute-optimal training, model size and the number of training tokens should be scaled equally, so each doubling of model size should be matched by a doubling of training tokens (Hoffmann et al., 2022).

To test this, they trained Chinchilla, a 70-billion-parameter model on 1.4 trillion tokens, with the same compute budget as the 280-billion-parameter Gopher, which had seen 300 billion tokens. Chinchilla outperformed Gopher across a broad range of downstream tasks and reached 67.5 percent average accuracy on the MMLU benchmark (Hoffmann et al., 2022). Because it had a quarter of the parameters, it was also cheaper to run afterward.

What "compute-optimal" means

A model is compute-optimal when, for a given training budget, its parameter count and training data are balanced so neither is wasted. Working from the paper's tables, the compute-optimal ratio comes out at roughly 20 tokens per parameter, which is exactly where Chinchilla sits (1.4 trillion divided by 70 billion). By that yardstick, GPT-3 at about 1.7 tokens per parameter and Gopher at about 1.1 were heavily undertrained. A smaller model trained on more data would have used the same compute more effectively.

One caveat matters. Compute-optimal means cheapest to train for a given quality, not cheapest overall. A model that will serve millions of users is run far more often than it is trained, so a lab may deliberately train a smaller model for longer than the Chinchilla ratio suggests, paying more up front to save on every later query. That tension returns in the inference section below.

Why GPUs, not CPUs, run this workload

A GPU (graphics processing unit) contains thousands of relatively simple processing cores that apply the same operation to many pieces of data at once, a design originally built to shade millions of pixels in parallel for video games. A CPU (central processing unit) has a handful of powerful cores tuned for fast sequential logic, branching decisions, and general-purpose tasks, work in which the next instruction depends on the result of the last one.

Matrix multiplication suits GPUs because every value in the output matrix can be computed independently of every other. Nothing has to wait. The operation can be split across thousands of cores running at the same time, whereas a CPU would work through far fewer pieces at once. Think of grading a stack of 10,000 identical multiple-choice tests. A single expert grader is fast, but 1,000 assistants each marking 10 tests finish sooner, because no test depends on another. When researchers began training deep neural networks on graphics hardware in the early 2010s, they found large speedups over CPUs. Our explainer on how GPUs work goes into the hardware.

Modern AI chips push this further with tensor cores, circuits built specifically for the multiply-and-accumulate math of matrix multiplication. NVIDIA's H100 product page lists these specifications for the SXM version:

Specification	H100 SXM
FP64	34 teraFLOPS
FP64 Tensor Core	67 teraFLOPS
FP32	67 teraFLOPS
TF32 Tensor Core	989 teraFLOPS (with sparsity)
FP16 Tensor Core	1,979 teraFLOPS (with sparsity)
FP8 Tensor Core	3,958 teraFLOPS (with sparsity)
GPU memory	80 GB
Memory bandwidth	3.35 TB/s
NVLink bandwidth	900 GB/s

NVIDIA marks the tensor-core rows as figures "with sparsity," meaning they assume a technique that skips a fraction of the arithmetic. Many training jobs do not benefit from it, so sustained rates are lower than the headline numbers.

The H100's 1,979 teraFLOPS FP16 tensor-core rating is about 30 times its 67 teraFLOPS FP32 rating, but the comparison flatters the chip. The larger number assumes sparsity and a lower-precision format built for matrix math, while the smaller number is general-purpose arithmetic (NVIDIA H100 product page).

The lesson is not that GPUs are magic. It is that the hardware is shaped like the workload. Lower-precision formats such as 16-bit and 8-bit floating point store each number in fewer bits, so a chip can do more of them per second and move more per unit of memory bandwidth, and neural networks tolerate the lost precision well. Our guide to LLM quantization covers that tradeoff.

The memory wall: why moving numbers costs as much as multiplying them

Raw arithmetic speed is only half the story. Every multiplication needs its inputs, and the inputs live in memory. If the chip can multiply faster than it can fetch numbers, it sits idle waiting. This is called being memory-bound.

Here is a back-of-envelope calculation for generating text from a 70-billion-parameter model. The model assumptions here are chosen for illustration: weights stored in 16 bits (2 bytes each), one user at a time, and no other overhead. The weights then occupy about 140 gigabytes. That does not fit on one 80-gigabyte H100, so suppose the weights are split across two, each reading 70 gigabytes per generated token. At the H100's 3.35 terabytes per second of memory bandwidth, reading 70 gigabytes takes about 21 milliseconds.

The arithmetic for the same token is much smaller: about 2 × 70 × 10⁹ = 1.4 × 10¹¹ operations. Even at a conservative 500 teraFLOP/s, that takes about 0.28 milliseconds. Fetching the weights takes roughly 75 times longer than multiplying them. The chip's math units are mostly waiting.

This is why serving systems batch many users together, so that one read of the weights serves many requests, and why techniques such as quantization help: smaller numbers mean fewer bytes to read. It also explains why chips pair their processors with very fast stacked memory, and why connections between chips, such as NVLink at 900 GB/s on the H100 SXM, matter. A cluster is only as fast as its slowest link.

Training versus inference: two different compute problems

Training requires two passes for every piece of data: the forward pass, which produces a prediction, and backpropagation, which computes how every weight should change based on how wrong that prediction was. Backpropagation costs roughly twice the forward pass, which is where the factor of six comes from.

Inference, using an already-trained model to answer a new prompt, requires only the forward pass. Per query it is far cheaper than training; there is no gradient calculation and no weight update. But inference compute scales with usage in a way training compute does not. A model is trained once and then queried by potentially hundreds of millions of people, and every query reruns the chain of matrix multiplications from scratch. For a deeper look at the two phases, see AI training vs. inference, and for the full training pipeline, how AI models are trained.

At product scale this adds up. A company running a popular chatbot does not pay a one-time bill and then serve requests for free. Every response triggers fresh computation proportional to the model's size and the length of the conversation. That is one reason providers push toward smaller, more efficient models for high-volume use, and why small language models have a real commercial role.

Does a longer context window increase compute?

Yes. Most transformer-based models compute self-attention across every pair of tokens in the current context (see how transformers pay attention to words for the mechanism), so in the naive case cost grows with the square of the context length. Production systems use many optimizations to soften that curve, and our piece on the context window explains the tradeoffs.

Reasoning models spend compute at answer time

A newer pattern adds a third place to spend compute. Rather than only training a bigger model, some systems let a model generate a long chain of intermediate reasoning before it answers, which means a single query can cost many times more than a plain reply. That approach is covered in test-time compute explained. It shifts some of the compute bill from training to inference, and it makes the per-query cost harder to predict.

How training compute has grown across model generations

The table below places several documented models side by side. Epoch AI's estimates for models whose developers have not published their training compute carry wider uncertainty.

Model	Year	Parameters	Estimated training compute (FLOP)	Source
GPT-3	2020	175 billion	3.14 × 10²³	Brown et al., 2020
Gopher	2021	280 billion	5.76 × 10²³	Hoffmann et al., 2022
Chinchilla	2022	70 billion	Same budget as Gopher	Hoffmann et al., 2022
GPT-4	2023	Undisclosed	about 2 × 10²⁵ (estimate)	Epoch AI, 2024
Gemini Ultra	2023	Undisclosed	about 5 × 10²⁵ (estimate)	Epoch AI, 2024

The clearest signal is the trend, not any single number. Dividing Epoch's GPT-4 estimate of about 2 × 10²⁵ by GPT-3's 3.14 × 10²³ gives roughly a 64-fold increase in three years. A steady 4-fold annual increase compounds to 64 over three years (4 × 4 × 4), so the two figures agree closely.

Growth of 4 to 5 times per year works out to training compute doubling about every five to six months (a calculation from Epoch's rate, not a figure Epoch states). For comparison, the exponential growth long associated with Moore's Law, which concerned transistor counts rather than training budgets, is much slower. The gap was filled by buying more chips, wiring them into bigger clusters, and running them longer, not by single chips getting four times faster each year.

The electricity bill and the data center

Compute is physical. Chips turn electricity into arithmetic and heat, and the heat must be removed. The International Energy Agency's 2025 report Energy and AI estimates that data centers used around 415 terawatt hours of electricity in 2024, about 1.5 percent of global electricity consumption. Its base case projects about 945 terawatt hours by 2030, just under 3 percent of global use, growing about 15 percent per year from 2024 (IEA, 2025).

The IEA ties much of that growth to AI. It projects accelerated servers, the machines built around GPUs and similar chips, to grow about 30 percent per year in its base case, compared with about 9 percent per year for conventional servers, and says they account for almost half of the net increase in data center electricity consumption (IEA, 2025). For a detailed look at where that electricity goes, read how AI data centers use electricity.

These are projections, not measurements, and they depend on assumptions about efficiency gains and demand. The IEA's base case is one of several scenarios.

Can efficiency break the trend?

If compute is so expensive, why not make the models cheaper? Researchers are doing exactly that, and several approaches reduce the cost of a given level of quality.

• Distillation trains a small "student" model to imitate a large "teacher," keeping much of the quality at a fraction of the serving cost. See AI model distillation.
• Mixture of experts (MoE) models route each token to only a few specialized sub-networks, so the model has many parameters but uses only some of them per token. See mixture of experts.
• Quantization stores weights in fewer bits, reducing memory traffic, as noted above.

These methods lower the cost per unit of capability, but they do not abolish the demand. When something gets cheaper, people often use more of it. Labs reinvest savings in larger runs or more queries, so total compute can keep rising even while efficiency improves.

There are also constraints on the other side. Epoch's frontier estimate of 4.2 times per year since about 2018 is slightly lower than its estimate for 2010 to 2024 as a whole, and any exponential trend meets limits in chip supply, electricity, and available training data. How those limits will bind is genuinely uncertain, and forecasts that claim certainty deserve skepticism.

Where the compute story goes wrong: common misconceptions

"New algorithms drive the compute bill." The core algorithm, a transformer made of attention and feedforward layers, was introduced in the 2017 paper "Attention Is All You Need." That paper reported results after training for 3.5 days on eight GPUs (Vaswani et al., 2017). What changed since is the scale at which labs run essentially the same idea, because the scaling-law findings say more compute reliably buys better models.

"More parameters always means a better model." Chinchilla complicated that. For a fixed budget, growing the dataset along with the model beat growing parameters disproportionately, and gave a model that was cheaper to run (Hoffmann et al., 2022). Treat parameter count and training compute as related but separate numbers.

"Training is the only cost that matters." Inference at scale is a recurring cost that never ends. A lab that builds a huge model to win a benchmark can end up with a product that is expensive to serve indefinitely.

"Bigger chips will solve it." Faster arithmetic helps, but memory bandwidth, chip-to-chip links, and power supply set limits too. The calculation above shows a chip can spend most of its time waiting for data.

"The headline FLOPS rating is what you get." Peak ratings depend on precision, sparsity, and ideal conditions. Real jobs sustain a fraction of peak, which is why the single-GPU illustration earlier used a figure far below the chip's rated maximum.

What this means for how AI gets built

The honest limitation in any compute discussion is that recent frontier models do not publish their training compute, so figures for GPT-4 and Gemini Ultra above are third-party estimates, not disclosures. What is well documented is the mechanism: matrix multiplication is nearly all of what neural networks do, GPUs are built to run that operation in parallel, memory movement limits how fast it can go, and two lines of published research (Kaplan et al., 2020, and Hoffmann et al., 2022) show that model quality improves predictably, if unevenly, with how compute is split between model size and training data.

The takeaway

AI's compute appetite is not a mystery born of complexity. It is the measurable cost of one simple operation, multiplied across billions of parameters and trillions of tokens, run on hardware shaped to parallelize exactly that operation, and then run again for every user who asks a question. The scaling-law research did more than describe the appetite. It changed how labs spend their budgets, shifting training away from ever-larger parameter counts and toward a balance with data. The best predictor of where AI costs go next is not a parameter count in a press release but the tokens-times-parameters arithmetic, the memory bandwidth to feed it, and the electricity to power it.

Frequently Asked Questions

Q: Why is matrix multiplication so central to AI compute demands? A: A neural network's forward pass and its backpropagation step are built almost entirely from matrix multiplications between data and learned weights. Repeating that across every layer, for every token, over a training set of hundreds of billions of tokens, produces the enormous total. GPT-3's training took about 3.14 × 10²³ operations (Brown et al., 2020).

Q: Why do GPUs work better than CPUs for AI training? A: Matrix multiplication splits into many independent calculations that do not wait on each other, so they can run simultaneously. GPUs have thousands of simple cores made for that parallel pattern, while CPUs have fewer, more powerful cores tuned for sequential logic. Modern AI GPUs add tensor cores built specifically for the multiply-and-accumulate math.

Q: Does running a trained model cost as much as training it? A: Per query, no. Inference skips backpropagation, so one response costs a small fraction of a training run. But a popular product answers enormous numbers of queries, and each one repeats the forward pass, so cumulative inference cost can become a large ongoing expense separate from the one-time training bill.

Q: What did the Chinchilla paper change? A: DeepMind's 2022 paper found that many large models were undertrained for their size, and that for a fixed compute budget, model size and training tokens should scale equally. Its 70-billion-parameter Chinchilla, trained on 1.4 trillion tokens, outperformed the 280-billion-parameter Gopher on the same budget (Hoffmann et al., 2022).

Q: How fast has training compute grown? A: Epoch AI estimates that training compute for notable models grew about 4.1 times per year from 2010 to May 2024, and about 5.3 times per year for frontier models, recommending 4 to 5 times per year as a baseline. Epoch's estimates put GPT-4 at roughly 2 × 10²⁵ FLOP, about 64 times GPT-3.

Q: Is a bigger model the same thing as more training compute? A: No. Training compute is roughly six times the parameter count times the number of training tokens. A 7-billion-parameter model trained on 2 trillion tokens uses more compute than a 30-billion-parameter model trained on 300 billion, even though the second sounds larger.

Q: How much electricity do data centers use? A: The International Energy Agency estimates about 415 terawatt hours in 2024, around 1.5 percent of global electricity use, and projects about 945 terawatt hours by 2030 in its base case. It attributes almost half of the net increase to accelerated servers, which are the AI-oriented machines.

Sources

• Brown, T. et al. (2020). "Language Models are Few-Shot Learners." https://arxiv.org/abs/2005.14165
• Kaplan, J. et al. (2020). "Scaling Laws for Neural Language Models." https://arxiv.org/abs/2001.08361
• Hoffmann, J. et al. (2022). "Training Compute-Optimal Large Language Models." https://arxiv.org/abs/2203.15556
• Vaswani, A. et al. (2017). "Attention Is All You Need." https://arxiv.org/abs/1706.03762
• Epoch AI (2024). "Training Compute of Frontier AI Models Grows by 4-5x per Year." https://epoch.ai/blog/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year
• NVIDIA, "H100 Tensor Core GPU" product page. https://www.nvidia.com/en-us/data-center/h100/
• International Energy Agency (2025). "Energy and AI: Energy demand from AI." https://www.iea.org/reports/energy-and-ai/energy-demand-from-ai

---

Read next:

• How transformers pay attention to words
• What small language models are good for
• How local AI tools protect privacy