AI glossary · Basic · Language models
Scaling laws
Scaling laws are empirical rules stating that as you increase an AI model's size (number of parameters), the amount of training data, and the compute used for training, model performance improves in a fairly predictable way.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Models & generation
- Fields
- Language models
- Added
- 2026-10-08
How it works
Researchers train several models of different sizes and measure how well each one predicts the next word (the loss value). When these results are plotted, a smooth curve appears in which error steadily decreases as model size, data and compute increase. Using this curve, you can roughly estimate the performance of a very large model before actually training it. It is also used to decide how to split a fixed compute budget between model size and data amount.
Why it matters
Training a single large model takes enormous cost and time. Scaling laws provide a basis for predicting how much performance gain such an investment will bring, and became one of the reasons AI companies invest heavily in GPUs and data centers. The rapid growth of today's LLMs is also closely tied to belief in these laws.
Things to know
Scaling laws are not laws of nature but trends obtained through observation. The gains from increasing scale get progressively smaller, and a shortage of high-quality training data is also cited as a problem. There is also no guarantee that a lower loss value is the same as an improvement in ability that people actually notice. So recently, besides scaling up pretraining, interest has grown in other ways to boost performance, such as post-training and test-time compute.
Example
When AI companies announce a new model and emphasize that it was trained with more GPUs and more data than before, that is a strategy premised on scaling laws. Before starting full-scale training, companies experiment with several small models to draw the curve, and based on it, decide the final model's size and amount of training data.