Scaling laws

Scaling laws are empirical power-law relationships showing that a language model's loss falls predictably as parameters, training data, and compute increase. They converted frontier model development from artisanal experimentation into capital planning: labs could forecast the return on a training run before spending on it, which is the economic logic behind every giant run on this wiki.¹

Kaplan and Chinchilla

Kaplan et al. (2020, OpenAI) fit the original power laws and drew a fateful conclusion: at fixed compute, bigger models trained on relatively less data are optimal. GPT-3 embodied that advice. Hoffmann et al. (2022, DeepMind, now Google DeepMind) redid the analysis with better learning-rate handling and found the opposite balance: compute-optimal training wants roughly twenty tokens per parameter, meaning the GPT-3 generation was substantially undertrained for its size. Their Chinchilla model, a quarter the size of Gopher but trained on four times the data, outperformed it across benchmarks including MMLU, and "Chinchilla-optimal" became the industry's planning default.¹²

After Chinchilla

Practice then diverged from compute-optimality: Meta AI's Llama line deliberately overtrained small models far past twenty tokens per parameter, because inference cost, not training cost, dominates a deployed model's economics. The 2024-2025 reasoning era added a second axis, inference-time scaling: performance improving with longer chains of thought at answer time, formalized in the o1-generation work of OpenAI and the open replications of DeepSeek. Whether pretraining scaling has hit diminishing returns is a live, contested question; proponents of continued scaling and skeptics both claim recent evidence, and this wiki records the debate rather than adjudicating it.³

Adoption

Scaling analysis governs the Transformer runs of every lab from Anthropic to the open publishers, sets the compute thresholds used in frontier-model regulation, and underlies the training-compute estimates on this wiki's hardware pages.¹³ The paradigm also developed a public literature: OpenAI's 2018 AI and Compute analysis documented the 3.4-month doubling of training compute,⁴ Richard Sutton's 2019 essay The Bitter Lesson supplied its philosophy,⁵ Gwern Branwen's 2020 The Scaling Hypothesis applied it to GPT-3,⁶ and Leopold Aschenbrenner's 2024 Situational Awareness extrapolated the trendlines into end-of-decade forecasts.⁷

References