Scaling laws of LLMs

One of the more consequential discoveries in modern AI research is that language model performance follows surprisingly predictable power laws. Get the right variables — compute budget, dataset size, parameter count — and you can forecast a model's loss with reasonable accuracy before spending a single GPU hour. These laws have reshaped how labs allocate billion-dollar training budgets and have a real say in where the field heads next.

The original scaling laws: Kaplan et al. (2020)

The story starts with a 2020 OpenAI paper by Kaplan, McCandlish, Henighan, and others. They trained language models from tens of thousands to billions of parameters and found that test loss (in nats) follows a power law in three variables:

N — number of model parameters (excluding embeddings)
D — size of the training dataset (in tokens)
C — compute used for training (in FLOPs)

The relationships:

L(N) \propto N^{-0.076} \qquad L(D) \propto D^{-0.095} \qquad L(C) \propto C^{-0.050}

In plain terms: every 10x increase in compute reduces loss by a fixed, predictable amount. The curves are smooth, span multiple orders of magnitude, and showed no sign of plateauing in the ranges tested.

A big takeaway from Kaplan was that model size matters more than data size. Their analysis said for a fixed compute budget you should scale parameters aggressively and train on relatively less data. Concretely: 10x your compute, increase model size by roughly 5.5x, increase training tokens by about 1.8x.

That finding directly influenced GPT-3 (175B parameters, ~300B tokens) and kicked off the "bigger is better" era.

Chinchilla rewrites it: Hoffmann et al. (2022)

Two years later, DeepMind's Hoffmann et al. published the Chinchilla paper, which fundamentally revised the optimal split between parameters and data.

They trained over 400 models from 70M to 16B parameters on varying amounts of data and found that Kaplan's models were undertrained. The compute-optimal strategy is to scale parameters and training tokens in roughly equal proportion.

Their revised rule of thumb: training tokens should be roughly 20x the number of parameters.

Model	Parameters	Kaplan-Optimal Tokens	Chinchilla-Optimal Tokens
1B	1B	~25B	~20B
10B	10B	~45B	~200B
70B	70B	~80B	~1.4T
175B	175B	~130B	~3.5T

Compute-optimal training tokens by scaling-law recipe. Y-axis is logarithmic — at 175B parameters, Chinchilla calls for 27× more training data than Kaplan does, which is why GPT-3 looks dramatically undertrained in hindsight.

The gap is huge at large scales. By Chinchilla's math, GPT-3 was severely undertrained — it should have seen roughly 3.5 trillion tokens rather than 300 billion. DeepMind proved it by training Chinchilla (70B parameters, 1.4T tokens) to outperform the much larger Gopher (280B parameters, 300B tokens) on the same compute budget.

Why the disagreement

The two teams reached different conclusions mostly because of methodology:

Learning rate schedules: Kaplan used a fixed cosine schedule across runs. Chinchilla tuned per run, which matters because smaller models with more data benefit from longer warmups.
Training budget allocation: Kaplan ran many models to partial convergence. Chinchilla varied both axes (params and tokens) across the full matrix.
Embedding parameters: Kaplan excluded them from N. Chinchilla included them, which shifts the curve for smaller models.

The consensus today leans Chinchilla, though the precise exponents are still being argued about. Meta's LLaMA models (2023-2024) were explicitly trained in the "Chinchilla-optimal" or even over-trained regime — LLaMA 7B trained on 1T tokens, well beyond its compute-optimal point, because inference cost scales with parameter count, not training tokens.

Compute-optimal training in practice

The practical version is an optimization problem. For a fixed compute budget, find the parameter/token pair that minimizes loss. Two approximations do the heavy lifting: a transformer's training compute is roughly $C \approx 6ND$ FLOPs, and the Chinchilla-optimal data budget is $D \approx 20N$ tokens. Substitute one into the other and the whole recipe falls out:

# Simplified compute-optimal calculation
# C ≈ 6 * N * D (approximate FLOPs for a transformer forward+backward pass)
# Chinchilla optimal: D ≈ 20 * N

def compute_optimal_config(flops_budget):
    # From C = 6 * N * D and D = 20 * N:
    # C = 6 * N * 20 * N = 120 * N^2
    N = (flops_budget / 120) ** 0.5
    D = 20 * N
    return int(N), int(D)

# Example: 1e24 FLOPs budget
params, tokens = compute_optimal_config(1e24)
# params ≈ 2.9B, tokens ≈ 57B

In reality, labs often deviate from compute-optimal on purpose. Over-training a smaller model is common when you care about inference efficiency — training is a one-time cost, inference happens millions of times. That's why LLaMA 3 8B was trained on 15T tokens, roughly 40x beyond Chinchilla-optimal.

Beyond loss: emergent capabilities

Scaling laws predict smooth improvement in cross-entropy loss. Real-world model behavior is less smooth. Research from Google and others has documented emergent capabilities — abilities that appear suddenly when models cross certain scale thresholds.

Few-shot arithmetic, chain-of-thought reasoning, and multi-step logical deduction all seem to "turn on" at particular scales rather than improving gradually. That makes scaling laws necessary but not sufficient for predicting what a model can actually do. Loss is continuous. Usefulness is often a step function.

What scaling laws predict for the future

If the current power laws continue to hold, a few things follow:

Data walls are real. High-quality text on the public internet is estimated at 5-15 trillion tokens. Models are already hitting that ceiling, which is what's driving interest in synthetic data and multimodal training.
Compute requirements keep growing. Halving loss from current frontier models would take roughly 1000x more compute, which is why labs are investing in 100,000+ GPU clusters.
Diminishing returns kick in. Power-law exponents are less than 1, so each successive improvement costs disproportionately more. Going from GPT-3 to GPT-4 quality required roughly 10-100x the compute. The next equivalent jump will cost even more.
Architecture improvements can shift the curve. Scaling laws describe a given architecture family. Innovations like mixture-of-experts, state-space models, or better attention mechanisms can shift the whole curve, hitting the same loss with less compute.

Scaling laws moved AI training from art toward engineering. They don't tell you everything — nothing about alignment, safety, or which capabilities will emerge — but they give you a useful lens for reasoning about the economics and trajectory of AI development. If you're making decisions about model training, infrastructure spend, or the direction of an AI product, you need to know them.