AI glossary · Advanced · Data & evaluation
Data contamination
Data contamination is when the test questions used to evaluate an AI model (benchmarks) end up in the model's training data, inflating its score beyond its real ability.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Foundations & applications
- Fields
- Data & evaluation
- Added
- 2026-10-08
How it works
LLMs are pretrained on a huge amount of text collected from the internet. But benchmark questions and answers are often published online too, through papers, GitHub, blogs and so on. If these are swept up when training data is collected, the model has effectively seen the test in advance. It can then get answers right by remembering them rather than solving the problems. In most cases developers did not add them on purpose; with so much data, they slip in unnoticed.
Why it matters
Benchmark scores are a main basis for comparing which model is better. If there is data contamination, these scores cannot be trusted, and the model may not perform as well as expected in real services. There is also a risk that researchers misjudge how well a new technique works. That is why whether contamination was checked is becoming an increasingly important trust criterion when models are announced.
Things to know
Contamination is hard to prevent completely. A common method is to find and remove parts of the training data that overlap with benchmark text, but slightly reworded or translated versions are easy to miss. So other methods are used alongside it: benchmarks that keep their questions private, evaluation with new questions created after the model was trained, and having the model re-solve questions with the numbers or wording changed. A large score gap between the original questions and slightly altered ones is grounds to suspect contamination.
Example
When a new LLM is announced with a high score on a well-known math benchmark, researchers often have it re-solve the same types of problems with only the numbers and wording changed. If the score drops sharply on the altered problems, they suspect it may have memorized the originals, in other words data contamination. That is why several AI companies and research teams publish contamination check results when they release models, or report separate scores evaluated on unpublished questions.