latent
A model with 70 billion parameters beat one with 280 billion while using the same training compute budget. DeepMind's Chinchilla result made a practical question hard to ignore: how much of your budget should buy a larger model, and how much should buy more training for it? Hoffmann et al. (2022)
The answer changes again when that model has to serve millions of requests. Spending more on training a smaller model can reduce the cost of every answer afterward. Scaling laws help reason about both decisions, provided we say what we're trying to optimize.
The original scaling laws: Kaplan et al. (2020)
A language model learns by predicting tokens, the pieces of text it processes. Its loss measures how poorly its predicted probabilities match the text it is tested on. Lower loss means better predictions on that test distribution; it isn't a direct score for intelligence or usefulness.
Kaplan and colleagues found smooth relationships between loss and model size, dataset size, and training compute. Their fitted relationships included:
Here, N counts non-embedding parameters, D counts training tokens, and C counts compute. The first two fits describe regimes where the other resources aren't the bottleneck. The compute exponent shown is their estimate for compute-efficient training. These are empirical fits within tested conditions, not universal constants.
The small exponents explain the economics. Under that compute fit, multiplying compute by 10 multiplies loss by about 0.89: roughly an 11% reduction, not a fixed number of points. Each comparable percentage improvement requires another multiplication of the budget.
Kaplan's allocation favored increasing model size faster than training data. A 10x compute increase implied about 5.4x as many parameters and 1.9x as many training tokens. The attraction was predictable improvement without testing every possible large model. The risk was choosing the wrong allocation curve.
Chinchilla changes the allocation
Hoffmann's team trained more than 400 models, from 70 million to over 16 billion parameters, varying their training budgets. Their result favored scaling model size and training tokens in roughly equal proportion. A useful shorthand became about 20 training tokens per parameter.
The large test was Chinchilla: 70B parameters and 1.4T training tokens, compared with Gopher's 280B and 300B. Chinchilla outperformed Gopher across the reported evaluation suite. Giving the smaller model more training was a better use of that compute budget. Training Compute-Optimal Large Language Models
The table and chart below retain the article's saved comparison inputs. The Chinchilla column follows 20 × parameters. The Kaplan column has no documented derivation in the saved data and should not be used as a training recommendation.
| Parameters | Saved Kaplan-labeled tokens (unverified) | Tokens from the 20:1 rule |
|---|---|---|
| 1B | ~25B | ~20B |
| 10B | ~45B | ~200B |
| 70B | ~80B | ~1.4T |
| 175B | ~130B | ~3.5T |
Archived comparison inputs on a logarithmic scale. The Chinchilla series uses 20 tokens per parameter; the Kaplan-labeled series lacks a documented derivation. Its 175B bars differ by about 27x, but that ratio is not a verified comparison of the papers' prescriptions.
For a hypothetical 175B model, the 20:1 shorthand gives 3.5T tokens. That calculation illustrates the extra data implied by the rule. It doesn't establish a universal optimum for every 175B model, since the dataset and training recipe also matter.
Why the methods disagreed
Both teams were estimating the best allocation from smaller experiments. How those experiments were run affected the answer.
Hoffmann and colleagues identified a learning-rate scheduling issue: a model partway through a long training schedule can look worse than one trained for that shorter duration with an appropriately matched schedule. Reading intermediate losses from the long run can therefore distort the allocation estimate. Their experiments varied training duration and matched the schedules to it. Methodology discussion
That is why I treat a scaling fit like a capacity-planning model. It helps choose where to spend the next unit of compute, but its assumptions travel with it. A changed data mixture or training procedure calls for checking the fit again.
Turning the rule into a budget
For a dense transformer, a common approximation is:
Combine it with the shorthand:
The arithmetic gives a starting configuration:
# Simplified compute-optimal calculation
# C ≈ 6 * N * D (approximate FLOPs for a transformer forward+backward pass)
# Chinchilla rule of thumb: D ≈ 20 * N
def compute_optimal_config(flops_budget):
# From C = 6 * N * D and D = 20 * N:
# C = 6 * N * 20 * N = 120 * N^2
N = (flops_budget / 120) ** 0.5
D = 20 * N
return int(N), int(D)
# Example: 1e24 FLOPs budget
params, tokens = compute_optimal_config(1e24)
# params ≈ 91.3B, tokens ≈ 1.83TThis is an illustrative calculation, not a measured training run. FLOPs count arithmetic operations; converting them into GPU hours requires hardware throughput and utilization. Attention costs, architecture details, and data quality also sit outside this little function. Its value is making the budget tradeoff visible before a more detailed estimate.
The bill continues after training
Training-optimal means getting the lowest loss for a fixed training budget. A deployed service also pays to run the resulting model.
Think of a hypothetical document classifier that will process millions of documents. A smaller model might need more training to reach the required quality, yet cost less per document afterward. Whether that extra training pays back depends on traffic and the actual serving costs. Context length, batching, hardware, and output length all affect those costs.
The original LLaMA paper explicitly considered inference efficiency: its 7B model trained on 1T tokens, well beyond the simple 20:1 ratio. Meta later trained Llama 3 on over 15T tokens. For the 8B model, the shorthand would suggest 160B tokens; 15T is about 94 times that amount. This is extra training relative to a training-compute optimum, not evidence of harmful overfitting.
The distinction matters when someone calls a model "overtrained." More training can be an intentional purchase of better quality at a fixed model size. The deployment objective determines whether it was money well spent.
What the curves can and cannot predict
A smooth loss curve doesn't guarantee smooth progress on every task. Wei and colleagues documented apparently sudden improvements in capabilities such as arithmetic as model scale increased. Schaeffer and colleagues showed that some apparent jumps depend on the scoring method.
For example, a test that awards a point only when every digit is correct can hide gradual improvement until complete answers begin passing. That is a useful reason to inspect both the metric and the mistakes. Lower average loss can guide training decisions, while a document classifier still needs evaluation on the documents it will handle.
The same care applies to forecasts. A fitted curve assumes the needed data remains available and useful. Villalobos and colleagues' data-supply study models that constraint using explicit assumptions about data stocks and training demand. Synthetic data and multimodal inputs may change the supply, but counting additional tokens alone doesn't establish their value.
Architecture changes can also change the cost of reaching a given loss. Mixture-of-experts models, state-space models, and different attention mechanisms need their own measurements; the dense-transformer budget calculation above cannot settle their economics.
Before applying a scaling rule, specify the objective. If the constraint is training compute, estimate the model-and-data allocation that gives the best validated loss. If the model will serve a large workload, include the lifetime serving bill and the task quality it must achieve. Chinchilla made the first calculation harder to get wrong. The second explains why a builder can understand that result and deliberately train a smaller model much longer.