AI glossary · Basic · Classic ML, recsys & search · Data & evaluation
Data leakage
Data leakage is the mistake of letting information that would not be available at real prediction time, especially test data or clues to the answer, slip into the training process. When this happens, evaluation scores come out high, but performance drops sharply in the real service.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Data, ops & platforms
- Added
- 2026-10-09
How it works
In machine learning, data is split into a training set and an evaluation set, and the model's ability is measured on evaluation data it has never seen. But if evaluation data gets mixed into training, or a column (feature) containing information almost identical to the answer is included in the input, the model memorizes this 'shortcut' instead of the real pattern. For example, using 'whether a treatment was prescribed' as an input while predicting whether a patient has a disease amounts to peeking at information from after the diagnosis was already made. Getting the preprocessing order wrong, such as normalizing with a mean or range computed on the full dataset before splitting, also lets information from the evaluation data leak in bit by bit.
Why it matters
Data leakage happens silently with no error message, so it is hard to detect. Because scores actually come out very good, the team believes the model is well built and goes as far as deploying it. In the real environment the shortcut information is not there, so performance suddenly collapses, and only then is time and money spent finding the cause. That is why a 'too good' score is seen as a signal to be suspicious of before celebrating.
Things to know
Time-ordered data should not be shuffled randomly before splitting; train on the past and evaluate on the future. It is also important to split data in groups so that data from the same person or document does not end up in both the training and evaluation sets. Preprocessing parameters should be computed only on the training data and then applied to the evaluation data. In LLMs, cases where benchmark questions get mixed into training data are separately called 'data contamination', which is broadly the same family of problem.
Leakage prevention checklist
Splitting the data
- For time-ordered data, did you train on the past and evaluate on the future?
- Is the same person or document absent from mixing into both sides?
Inputs and preprocessing
- Are there no columns that would be unknown at prediction time?
- Were normalization parameters computed only on the training data?
Interpreting results
- Is the score suspiciously good?
Example
Suppose an online store builds a model to predict 'will this customer leave next month?' If a column like 'account deletion date' is accidentally included in the input, the model predicts almost perfectly in evaluation, but becomes useless in the real service, where it needs to find customers who have not left yet. In practice, teams prevent this kind of leakage by checking each input column one by one: 'is this information really available at the time of prediction?'