Scaling Laws Explained: Why Bigger Models Got Better

Neural scaling laws are the reason labs can plan enormous training runs before they start them. They say that a language model's error falls along a smooth, predictable curve as you add parameters, training data, and computing power, so a small experiment can forecast a much larger one. Two papers define the field: Kaplan and colleagues at OpenAI in 2020, and Hoffmann and colleagues at DeepMind in 2022, who found that the first recipe under-trained its models. This explainer covers what each result says, how they differ, and where the curves stop being reliable.

What a scaling law is

A scaling law is an empirical relationship: a measured quantity, usually the model's loss, changes with a resource in a regular way. Loss here means cross-entropy, the average "surprise" of the model at each real next token in a test text. Lower is better, and a loss of zero would mean the model predicted every token perfectly.

The regularity researchers found is a power law. If you plot loss against model size with both axes on logarithmic scales, the points fall close to a straight line. The slope of that line is the exponent, and it tells you how much each tenfold increase buys you.

Three resources, one currency

Three quantities matter. Parameters (N) are the learned numbers inside the network. Tokens (D) are the pieces of text the model trains on, as described in the explainer on how LLM tokenization works. Compute (C) is the total floating-point arithmetic the training run performs. They are linked, because training a bigger model on more tokens takes more compute, and a fixed budget forces a choice between them.

The idea predates large language models. In 2017 Joel Hestness and colleagues reported that generalization error follows a power law across many factors in machine translation, language modeling, image processing, and speech recognition, and noted that the exponents were not yet explained by theory. They also found that model improvements tended to shift the error curve without changing its exponent. That finding still shapes how researchers think: better architectures move the line down, while scale moves you along it.

An analogy: a toll road that gets steeper

A power law behaves like a road where every mile costs the same fraction more than the last. Early progress is cheap. Going from a thousand parameters to ten thousand is a small step on a log plot and gives a big visible drop in loss. But each further tenfold jump takes ten times the resources for the same proportional gain. The curve never flattens into a wall, and it never lets you stop paying either.

How researchers measure a scaling law

Nobody discovers a scaling law by thinking about it. They train many models and look at the results. A typical study builds a ladder of models that differ in size, trains each on varying amounts of data, records the final loss on a held-out text, and then fits a smooth function to the whole grid. The fitted function is the "law," and its value comes from predicting runs that were not in the grid.

The Chinchilla paper is a good illustration because it used three different procedures on the same family of runs and compared the answers. In its first approach, the authors fixed each model size, varied the number of training tokens, and took the lower envelope of all the training curves to see which size was best at each compute level. In its second, they fixed a total compute budget (an "IsoFLOP" profile) and varied model size, looking for the size that gave the lowest loss. In its third, they fitted a parametric equation for loss as a function of parameters and tokens to every run. If all three agree, the conclusion is more trustworthy. In the paper's results, the first two agreed closely and the third was a little different, which becomes relevant later.

Why small experiments can predict big ones

The whole enterprise depends on extrapolation. You train models that cost a few thousand dollars and use the fit to decide how to spend a budget a thousand or a million times larger. That is only safe if the power law keeps holding past the largest model you tested, which no theory guarantees. Kaplan's team reported that the trends held across many orders of magnitude, a very wide range, and Meta's Llama 3 team used the same method to choose a flagship size before training it. But the method is empirical, and an empirical rule is a bet on continuity.

Kaplan et al. 2020: three variables, three power laws

Jared Kaplan, Sam McCandlish, and eight co-authors at OpenAI posted "Scaling Laws for Neural Language Models" on January 23, 2020. Their abstract reports that cross-entropy loss follows a power law in model size, dataset size, and training compute, with some trends holding across more than seven orders of magnitude (the paper's introduction says more than six). They also found that architectural details such as network width or depth had minimal effects within a wide range, so the total parameter count mattered far more than the shape.

The specific numbers are worth seeing once. In the paper's summary, the loss as a function of the number of non-embedding parameters is L(N) = (N_c / N)^α_N, with α_N of about 0.076. The compute-efficient frontier has an exponent of about 0.050, and dataset size follows its own power law with a similar form. Each law holds when the other two resources do not bottleneck the model.

What an exponent of 0.076 means

The exponent is easier to read as a rule of thumb. Because loss scales as N raised to the power of −0.076, doubling the parameters multiplies the loss by 2 to the −0.076, which is about 0.949, a drop of roughly 5 percent. A tenfold increase in parameters multiplies it by about 0.84, a 16 percent drop. A hundredfold increase gives about 0.70.

Those are modest-looking percentages, but the loss is measured per token and the gains compound across every token the model writes. The point of the law was never that one doubling is dramatic. It is that you could fit the line on small models and predict, with surprisingly little error, what a much larger model would score.

The recommendation: go big, stop early

Kaplan's team drew two conclusions about spending a fixed budget. First, larger models are significantly more sample-efficient: a big model reaches a given loss with fewer training tokens than a small one. Second, for a fixed compute budget, the most efficient strategy is to train very large models on a relatively modest amount of data and stop well before convergence.

Their fitted allocation was that the best model size grows with compute raised to about the 0.73 power, while the data should grow only with about the 0.27 power. For a tenfold increase in compute, that rule means the model gets about 5.4 times bigger and the data grows only about 1.9 times. The Chinchilla paper observes that this is what happened in practice: most large models of the period were trained on approximately 300 billion tokens, however large they were.

Chinchilla 2022: the recipe was under-training

In 2022 Hoffmann and colleagues at DeepMind published "Training Compute-Optimal Large Language Models," which revised the answer. They trained more than 400 language models, ranging from 70 million to over 16 billion parameters, on 5 to 500 billion tokens, and fit the relationship between the three resources again.

Their conclusion was that for compute-optimal training, model size and training tokens should scale in equal proportion: every doubling of the model should be matched by a doubling of the data. The authors argued that existing large models were significantly undertrained, because recent work had scaled model size while keeping the data roughly constant at about 300 billion tokens.

An analogy: the oven and the flour

Picture a bakery with a fixed budget that has to buy an oven and flour. Kaplan's recipe said to buy the biggest oven you can and use only a little flour. Chinchilla said that approach wastes money: a mid-sized oven running at full capacity beats a giant one that is mostly idle. Gopher, DeepMind's earlier 280-billion-parameter model, was the giant oven. It saw roughly 300 billion tokens, about one token per parameter.

The test: Chinchilla versus Gopher

To check the new rule, DeepMind trained Chinchilla with 70 billion parameters on 1.4 trillion tokens, using the same compute budget as Gopher but about four times more data. Chinchilla outperformed Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a wide range of downstream tasks. It reached an average accuracy of 67.5 percent on the MMLU benchmark, more than 7 points above Gopher. Because it was smaller, it also needed less compute for fine-tuning and inference.

Dividing 1.4 trillion tokens by 70 billion parameters gives about 20 tokens per parameter. That ratio became the folk rule of Chinchilla scaling: for a compute-optimal run, plan on roughly 20 training tokens for every parameter. Note that the paper itself states the principle as "double the tokens when you double the parameters," and the 20-to-1 figure is arithmetic on the Chinchilla model rather than a headline claim.

Comparing the two recipes

The table below uses the exponents reported in the Chinchilla paper, where N_opt grows as compute to the power a and D_opt as compute to the power b. For Kaplan et al., the paper lists a = 0.73 and b = 0.27.

Recipe	a (model size)	b (data)	Model growth for 10x compute	Data growth for 10x compute
Kaplan et al. (2020)	0.73	0.27	about 5.4x	about 1.9x
Chinchilla, approach 1	0.50	0.50	about 3.2x	about 3.2x
Chinchilla, approach 2	0.49	0.51	about 3.1x	about 3.2x
Chinchilla, approach 3	0.46	0.54	about 2.9x	about 3.5x

The growth columns are calculated from the exponents. Source of exponents: Hoffmann et al. (2022).

Why the two papers disagreed

The Chinchilla authors identified a methodological reason. Kaplan et al., they write, used a fixed number of training tokens and a fixed learning-rate schedule for all models. The DeepMind team instead matched the learning-rate schedule to the number of training tokens, which they found gives the best final loss regardless of model size. A schedule that is too long for a small model, or too short for a large one, distorts the comparison, and that distortion nudged the earlier fit toward favoring size.

The Chinchilla result has itself been scrutinized. In 2024 Tamay Besiroglu and colleagues at Epoch AI attempted to replicate the third of the paper's three estimation methods. They found that the reported estimates conflicted with the first two methods, that the fit did not match the data extracted from the plots, and that the confidence intervals were implausibly narrow: intervals that tight, they argue, would need more than 600,000 experiments, while the original work likely ran fewer than 500. Their own re-derivation produced results consistent with the other two methods. The general lesson held up, but the third method's exact numbers did not.

A budget example you can check

A rough rule of thumb links the three resources: training compute is approximately six floating-point operations for every parameter for every training token, so C is about 6 times N times D. This is a widely used approximation, not an exact count, but it lets you sanity-check the Chinchilla numbers. For Chinchilla, 6 times 70 billion times 1.4 trillion is about 5.9 times 10 to the 23rd operations. For Gopher, 6 times 280 billion times 300 billion is about 5.0 times 10 to the 23rd. The Chinchilla paper lists Gopher's budget as 5.76 times 10 to the 23rd operations, so the approximation lands within about 13 percent (the paper's appendix gives a more careful figure of about 6.3 times 10 to the 23rd), and the two models' budgets are indeed close to each other.

The same arithmetic shows why the two models differ so much in character even though they cost the same. Spend the budget on 280 billion parameters, and the data you can afford is about 300 billion tokens. Spend it on 70 billion parameters, and you can afford four times as many tokens. Same bill, different split, and the second split gave the better model.

For the training pipeline these numbers feed into, see how AI models are trained, and for the architecture being scaled, the guide to how large language models work.

Compute-optimal is not the same as inference-optimal

A compute-optimal model minimizes loss for a given training budget. That is the right target if training is the only cost. For a model that will answer billions of queries, it is the wrong one, because every query costs compute proportional to the model's size.

Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle modified the Chinchilla framework to account for this in "Beyond Chinchilla-Optimal," presented at ICML 2024. Their finding is that researchers who expect large inference demand, on the order of a billion requests, should train smaller models on more data than the Chinchilla rule suggests. They trained 47 models to validate the formula and found that quality kept improving at token-to-parameter ratios up to 10,000, far beyond 20. They also note that scaling laws fitted on typical ratios overestimate the benefit of additional tokens at such extreme ranges.

What labs actually do

Public technical reports show the pattern. Meta's Llama 3 report says its models were pre-trained on a corpus of about 15 trillion multilingual tokens, compared with 1.8 trillion for Llama 2. The flagship has 405 billion parameters and was trained on 15.6 trillion text tokens, about 38 tokens per parameter. The report says its own scaling-law extrapolation suggested a 402-billion-parameter model on 16.55 trillion tokens, about 41 tokens per parameter, which the authors call approximately compute-optimal for their budget. The same report says the smaller models were trained for much longer than is compute-optimal. At 20 tokens per parameter, an 8-billion-parameter model would see about 160 billion tokens, roughly one-hundredth of the 15-trillion-token corpus.

Small, heavily trained models are cheap to serve, which is the logic behind techniques like model distillation. The scaling law tells you how to spend a training budget. It does not tell you how to spend a lifetime serving budget, and the best choice depends on how many people will use the model.

Emergent abilities and the "mirage" debate

Scaling laws predict loss, a smooth number. But people care about abilities: can the model do arithmetic, translate, write code? In 2022 Jason Wei and colleagues reported that some abilities appear to arrive abruptly. They defined an ability as emergent if it is not present in smaller models but is present in larger models, and argued that such abilities cannot be forecast by extrapolating from smaller ones.

A 2023 paper by Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo, "Are Emergent Abilities of Large Language Models a Mirage?", offered a different explanation. They suggested that apparent emergence comes from the choice of evaluation metric. A metric that gives credit only for a fully correct answer, such as exact match, can jump from near zero to high as the model improves, even when the model's per-token probabilities were improving smoothly all along. With linear or continuous metrics, the same models show gradual, predictable improvement. The authors tested this on the InstructGPT and GPT-3 families, in a meta-analysis of BIG-Bench, and by producing seemingly emergent abilities in vision tasks from the choice of metric alone.

How to read the dispute

The two papers are not obviously in conflict on the facts. Schaeffer and colleagues do not say that large models cannot do things small models cannot. They say that the shape of the jump depends on how you score it. For a user, the practical question is whether a particular task will work at a particular size, and the loss curve alone does not answer that. A model can be steadily improving on average and still cross the threshold for a task all at once. For ways to measure that, see the discussion of why AI benchmarks can be misleading.

The data wall and repeated epochs

Both papers treat data as something you can always get more of. Training-set size is one of the three variables, and the Chinchilla rule demands a lot of it: a compute-optimal 500-billion-parameter model would want roughly 10 trillion tokens at 20 per parameter, by simple arithmetic. The supply of high-quality text is finite.

Pablo Villalobos and colleagues at Epoch AI examined this in "Will we run out of data?" Their analysis concluded that if current trends continue, models would be trained on datasets about the size of the existing stock of public human-written text between 2026 and 2032, possibly somewhat earlier if models are overtrained. They suggest synthetic data, transfer learning from data-rich domains, and data-efficiency gains as ways to sustain progress.

Can you reuse the same data?

Niklas Muennighoff and colleagues tested one escape route in "Scaling Data-Constrained Language Models." Their study used about 400 training runs, up to 900 billion training tokens and models of up to 9 billion parameters. They found that training with up to four epochs of repeated data yields negligible changes in loss compared with unique data. Beyond that, the value of extra compute decays toward zero as repetition increases. They proposed a scaling law that accounts for the declining value of repeated tokens and excess parameters, and tested other mitigations such as adding code data and relaxing quality filters.

So the first few passes over the same text are nearly free, and later ones are not. Another route is generated text, covered in the explainer on synthetic data for AI training, though it brings its own risks and its own scaling questions.

The compute side: how fast budgets grew

Scaling laws reward spending, so it is useful to know how fast spending has grown. Epoch AI reports that the training compute used to create notable AI models has grown at about 4.4 times per year since 2010. A regression on 428 notable models gives a slightly higher estimate of 4.7 times per year, with a 90 percent confidence interval of 4.3 to 5.2, which is roughly a doubling every six months.

That growth is the practical context for the physical-world constraints: chips, electricity, and cooling. Training and serving these models draws real power, as the explainer on how AI data centers use electricity describes. A power law with a small exponent is a strong incentive to keep building, and a strong demand on infrastructure.

What scaling laws do not predict

The laws are narrow, reliable statements about one measurement under controlled conditions. It helps to list what lies outside that.

The laws predict	The laws do not predict
Test loss on held-out text from the same distribution	Whether a specific skill appears at a specific size
How loss changes as N, D, or C change, within the fitted range	Behavior far outside the fitted range
Rough optimal split of a training budget	Optimal split when inference cost, data quality, or data supply matter
Gains from more of the same recipe	Gains from changing the recipe (new architectures, new data mixes)

A scaling law is a map of one recipe, not a forecast of all progress.

Quality and recipe

The curves assume a fixed kind of data and a fixed training recipe. Kaplan's team found that network shape mattered little, but they did not claim that data quality or training method did not. Hestness and colleagues observed that model improvements shift the curve rather than change its slope, which means a better recipe changes the intercept. Two labs spending the same compute can land at different losses.

Beyond pretraining loss

Another limit is that pretraining loss is only part of what makes a deployed model useful. Post-training steps and extra computation at answer time also change what a model can do, as the article on test-time compute and LLM reasoning explains. That opens a second axis along which spending can be traded for quality, one that the original pretraining laws do not cover. The wider field is mapped in the Complete AI Guide and the Complete Machine Learning Guide.

Extrapolation risk

Every fit has a range. The Kaplan paper's trends held across six to seven orders of magnitude, which is wide, but frontier models sit at the edge of that range or beyond it. The Sardana paper's warning, that laws fitted at typical token-to-parameter ratios overestimate gains at extreme ratios, is a concrete example of an extrapolation failing. The Besiroglu replication is another: reported confidence was far tighter than the experiments could support.

Three common misreadings

Scaling laws are often summarized in a sentence, and the sentence usually loses something.

"Just make it bigger." This was close to the advice of the 2020 paper, and it is not the advice of the 2022 one. At fixed compute, adding parameters without adding data wastes money, which is the entire Chinchilla finding. Bigger helps only when everything else keeps pace.

"Every model should have 20 tokens per parameter." The ratio comes from one family of experiments and answers one question: how do I minimize training loss for a given training budget? It does not answer how to minimize total cost over the model's lifetime. The Llama 3 flagship trained at roughly 38 to 41 tokens per parameter, and the smaller Llama 3 models went far beyond that, because inference cost matters to a lab that serves them.

"A smooth curve means smooth capabilities." The curve describes the average prediction error per token. How that error maps onto a specific skill depends on the skill and the metric, as the emergence debate shows. A smooth line underneath can still produce a model that suddenly passes a test it failed the previous month.

The takeaway

The scaling-law story is less about bigness than about predictability. Kaplan et al. showed that language-model loss follows power laws in parameters, data, and compute, which turned model building into something closer to engineering than guesswork. Hoffmann et al. corrected the allocation: with a fixed budget, a smaller model trained on far more data, about 20 tokens per parameter, beat a much larger one trained on little. Later work added the complications. Serving costs push labs toward even smaller, longer-trained models, the supply of fresh text is finite, apparent jumps in ability may depend on how they are scored, and every fitted curve is only as trustworthy as its range. The laws remain the best tool for planning a training run, and a poor tool for predicting what a model will be able to do.

Frequently asked questions

Q: What is a neural scaling law? A: A neural scaling law is an empirical rule that a model's test loss falls as a power law in parameters, training data, or compute. Kaplan et al. (2020) found such trends across six to seven orders of magnitude for language models, with a parameter exponent of about 0.076.

Q: What is the Chinchilla scaling law? A: It is the finding by Hoffmann et al. (2022) that, for a fixed compute budget, model size and training tokens should grow in equal proportion. Their 70-billion-parameter Chinchilla, trained on 1.4 trillion tokens, outperformed the 280-billion-parameter Gopher, which used the same compute. That works out to about 20 tokens per parameter.

Q: How did Chinchilla differ from the Kaplan scaling laws? A: Kaplan et al. recommended mostly growing model size, with the best size scaling as compute to the 0.73 power. Chinchilla found exponents near 0.5 for both size and data. The authors attribute part of the gap to Kaplan's use of a fixed learning-rate schedule and token count across models.

Q: Does a bigger model always perform better? A: Not at fixed compute. Chinchilla showed that a smaller model trained on more data can beat a larger one trained on less. Past a point, the added cost of serving a large model also favors smaller models trained longer, as Sardana et al. (2024) show.

Q: What are emergent abilities in language models? A: Wei et al. (2022) define an ability as emergent if it is absent in smaller models but present in larger ones. Schaeffer et al. (2023) argue that many apparent jumps come from the choice of metric and that continuous metrics show smooth improvement.

Q: Will AI models run out of training data? A: Epoch AI researchers estimated in 2022, revising in 2024, that models could be trained on datasets the size of the entire public human-text stock between 2026 and 2032. Repeating data for up to four epochs costs little, according to Muennighoff et al., and synthetic data is another option.

Q: How fast is the compute used to train AI models growing? A: Epoch AI reports about 4.4 times per year for notable models since 2010, with a regression estimate of 4.7 times per year across 428 models. That is a doubling roughly every six months.

Sources

• Scaling Laws for Neural Language Models (Kaplan et al., arXiv:2001.08361): the 2020 OpenAI paper; exponents from the ar5iv HTML version.
• Training Compute-Optimal Large Language Models (Hoffmann et al., arXiv:2203.15556): the Chinchilla paper; exponents and token counts from the ar5iv HTML version.
• Chinchilla Scaling: A replication attempt (Besiroglu et al., arXiv:2404.10102): critique of the third estimation method.
• Beyond Chinchilla-Optimal (Sardana et al., arXiv:2401.00448): inference-aware scaling, ICML 2024.
• Emergent Abilities of Large Language Models (Wei et al., arXiv:2206.07682): the definition of emergence.
• Are Emergent Abilities of Large Language Models a Mirage? (Schaeffer et al., arXiv:2304.15004): the metric-choice argument.
• Scaling Data-Constrained Language Models (Muennighoff et al., arXiv:2305.16264): repeated-epoch results.
• Will we run out of data? (Villalobos et al., arXiv:2211.04325): the 2026 to 2032 estimate.
• Deep Learning Scaling is Predictable, Empirically (Hestness et al., arXiv:1712.00409): the 2017 precursor.
• The Llama 3 Herd of Models (arXiv:2407.21783): token counts and the scaling-law extrapolation for the flagship.
• Epoch AI, training compute trend: the 4.4 times per year growth estimate.