Scaling laws

Training · 2022

The idea

Loss falls predictably as model, data and compute grow together - predictably enough to plan billion-dollar runs around.

The growth history, anchored: GPT-2 (2019) was 1.5B parameters, a 1,024-token window, ~100B training tokens. Llama 3.1 (2024): 405B parameters trained on 15T tokens, with modern windows running from hundreds of thousands toward a million tokens (ch. 0:32-0:46).

Two papers, two years, one badge. The badge above says 2022 on purpose - this page’s verdict (the model/data split) is Hoffmann et al.’s finding, not Kaplan et al.’s. They disagree, and the disagreement is the actual history:

PaperScaling Laws for Neural Language ModelsKaplan, McCandlish, Henighan et al. · OpenAI/JHU · 2020arXiv:2001.08361 →

Kaplan (2020) established the power-law relationship first - loss falls predictably with scale - but its own prescription for a 10x compute budget was to grow the model 5.5x and the data only 1.8x. Following that advice is why GPT-3-era models were, in retrospect, undertrained.

PaperTraining Compute-Optimal Large Language ModelsHoffmann, Borgeaud, Mensch et al. · DeepMind · 2022arXiv:2203.15556 →

Hoffmann et al. - Chinchilla - corrected it two years later: model size and training tokens should scale in equal proportion. Their 70B Chinchilla, trained on 4x Gopher’s data, beat the 280B Gopher outright. That equal-split idea is the “best split between model size and data” this page’s verdict describes.

One honesty note: the GPT-2-to-Llama-3.1 anchors above show how far scale-up went, not the compute-optimal tradeoff curve itself - that curve is the two papers above, not this page’s illustrative frontier.

Verdict

What it fixedSizing became a calculation: for a given compute budget there is a best split between model size and data.

What it still costsThe laws are empirical fits, not physics - data quality and regime changes bend the curves.

The story, animated

GPT-2 to Llama 3.1, on the same illustrative frontier - how far scale-up went, and why the curve itself is a teaching shape, not a fitted law.

Where it sits

Worth having first: Pretraining

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).