AI glossary · Beginner · Language models
Pre-training
Pre-training is the first training stage, in which an AI model reads a very large amount of data such as web text, books and code to learn basic knowledge about language and the world. The model that finishes this stage is then refined for various uses.
About this term
- Level
- Beginner · Core ideas for first-time readers
- Category
- Basics
- Fields
- Language models
- Added
- 2026-10-08
How it works
In LLM pre-training, the model usually practices, an enormous number of times, predicting the next token (a piece of text) after seeing the beginning of a sentence. The answer is already in the original text, so people do not need to label each answer by hand. Each time the model is wrong, it slightly adjusts its internal parameters (the numbers that hold what the model has learned), and in this process it also picks up things like grammar, factual knowledge and traces of reasoning. Image and video models likewise first learn basic patterns from large amounts of data in a similar way.
Why it matters
Most of a model's basic abilities and knowledge come from pre-training. A well pre-trained model can easily be adapted to different tasks such as translation, summarization and coding, which is why it is called a 'foundation model.' As a result, the amount and quality of data, the model size and the amount of compute used strongly determine performance, and the research on this relationship is called scaling laws.
Things to know
Pre-training takes a very large amount of compute such as GPUs, so in many cases only large companies or research labs can do it from scratch. A model that has only been pre-trained is better at continuing text than at answering questions helpfully, so it needs additional post-training to be used in a conversational service. Errors and biases in the training data can also remain in the model as they are. In addition, the model does not know about events after the point when its training data was collected.
Example
The LLMs behind the conversational AI services we use have all first gone through pre-training on large amounts of text and were then refined for conversation. Foundation models that generate video and audio together, such as 'Kandinsky 6.0 Video' covered on this site, broadly follow the same flow: first learn basic abilities from lots of data, then adjust for the intended use.