AI glossary · Advanced · Data & evaluation · Language models
Data filtering
Data filtering is the work of picking out and discarding low-quality, harmful or useless items from collected data before training an AI model. What you leave out shapes model performance as much as what you put in.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Foundations & applications
- Fields
- Data & evaluation, Language models
- Added
- 2026-10-09
How it works
Data usually passes through several filters in sequence. First, the language of each text is identified, and simple rules filter out text that is too short or broken, or that is advertising or spam. Next comes deduplication, which removes text that appears multiple times. Finally, a small classifier model or an LLM trained to recognize 'good text' assigns scores, and only items above a certain score are kept. A step that finds and removes personal information or harmful content is also included.
Data shrinking at each stage
Why it matters
Raw data scraped from the internet contains a lot of meaningless text, repeated phrases and incorrect information. Training on this data as-is makes the model pick up odd styles of speech and only raises training costs. As the industry widely recognized that training on well-filtered data can produce a smarter model with the same compute, filtering became a core competitive factor in pre-training.
What to keep in mind
Results differ depending on who defines 'good data.' If only textbook-like text is kept, colloquial language or the speech of particular regions or groups may disappear, and the model can become skewed. Filtering is also responsible for preventing data contamination, where evaluation questions slip into training data. So how strongly to filter is adjusted while checking the effect through experiments.
Example
Companies building large language models commonly collect public web data, score it with a quality classifier and use only high-scoring data for pre-training. Some public training datasets also release a separate version containing only 'text with high educational value' selected by an LLM. When a company builds a chatbot on internal documents, it also needs to filter out outdated documents, drafts and duplicates first to improve answer quality.