Deep dive
Scaling laws explained
The empirical finding behind much of modern AI strategy.
Scaling laws describe how a neural network's performance changes as you scale up the resources used to train it. They are one of the empirical foundations behind modern AI strategy, and a reason Dario Amodei treats AI progress as something that can be forecast.
Scaling Laws for Neural Language Models (2020)
The 2020 paper, with Amodei among the authors, found that language model loss falls as a smooth power law as model size, dataset size and training compute increase. Larger models also proved more sample-efficient. The result suggested that spending more compute would reliably buy better models.
AI and Compute (2018)
Two years earlier, AI and Compute documented that the compute used in the largest AI training runs had been doubling roughly every 3.4 months since 2012, a rise of about 300,000 times by 2018.
Refinements
Later work from other labs, notably DeepMind's 2022 “Chinchilla” study, refined how to balance model size and training data for a fixed compute budget. The central lesson, that scale yields predictable gains, has held up even as the details changed.
Why it shaped Amodei's outlook
- Forecasting. Predictable curves make it possible to anticipate capabilities.
- Urgency. If progress continues, safety research and policy must keep pace.
- Cost. Frontier training is expensive, which is why chips and compute feature in his policy arguments. See On DeepSeek and Export Controls.
Limits and debate
Scaling laws are empirical regularities, not laws of nature. Researchers debate how long they will hold, what they imply for particular skills, and how much new ideas matter relative to raw scale.