AI glossary · Advanced · Data & evaluation · Language models
Data mix
A data mix is the combination that sets in what proportions different kinds of data, such as web documents, code, books, math and multiple languages, are blended when training an AI model. Even with the same amount of data, a model's abilities change depending on the mixing proportions.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Foundations & applications
- Fields
- Data & evaluation, Language models
- Added
- 2026-10-09
How it works
First, training data is grouped by source or type, for example general web text, code, academic papers, math problems, conversations and documents in other languages. Then a proportion is set for how often each group is shown during training. Data that is high quality but small in volume is sometimes shown repeatedly, while web data that is large in volume but low in quality may be reduced. Changing the proportions by training stage, raising the share of high-quality data toward the end, is also used.
Why it matters
Because pretraining takes enormous compute costs, a data mix is hard to undo once set. Raising the share of code can improve coding and logical reasoning but may make other abilities relatively weaker, so proportions are always a trade-off. That is why many research teams first experiment with several proportions on small models and apply the combinations that work well to large models. Now that model architectures have become similar, the data mix is considered a key factor behind performance differences between models.
The loop for finding a data mix
Finding good proportions
- 1Set proportions
- 2Train small model
- 3Evaluate abilities
- 4Adjust proportions
- 1. Set proportions Web, code, math, etc.
- 3. Evaluate abilities Benchmarks, etc.
- 4. Adjust proportions Strengthen weak abilities
Things to know
There is no guarantee that proportions that worked on a small model will work just as well on a large one. Also, adjusting proportions based only on benchmark scores can produce a model that is strong only on specific tests. Most companies do not disclose the exact data mix of their models, which makes outside comparison difficult. Its effect can only be judged properly alongside cleaning work such as data filtering and deduplication.
Example
Research teams building open-weight LLMs often explain in their technical reports the principles they used to mix web documents, code, math and multilingual data. For example, a model that emphasizes coding ability raises the share of code data, and when building a model that is good at a particular language such as Korean, teams deliberately increase the share of documents in that language. For national models promoted as sovereign AI, how much native-language data to mix in also becomes a key design decision.