AI glossary · Advanced · Data engineering · Language models
Deduplication
Deduplication means keeping only one copy and filtering out the rest when identical or nearly identical documents, sentences or images appear multiple times in training data. It is an essential step, especially when cleaning LLM training data collected in bulk from the internet.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Data, ops & platforms
- Fields
- Data engineering, Language models
- Added
- 2026-10-08
How it works
First, exact duplicates, which do not differ by a single character, are found. A hash (a value that turns the content into a short unique number) is computed for each document, and documents with the same number are treated as the same document, keeping only one. Next come near-duplicates that differ only in, say, ad text or dates. Techniques such as MinHash, which splits documents into short chunks of characters and quickly estimates how much they overlap, or methods that use embeddings to find documents with similar meaning are used. A threshold (how similar documents must be to count as duplicates) is set, and only one representative is kept from each group that is at least that similar.
Steps for filtering out duplicates
Why it matters
Internet data contains a huge amount of republished articles, copied code and repeated terms of service. If these duplicates are left in, the model sees the same text many times, making it more likely to memorize those passages whole and reproduce them verbatim, which can cause privacy or copyright problems. Training compute is also wasted on the same content. In addition, when evaluation benchmark questions get mixed into the training data, data contamination occurs and inflates measured ability; filtering out overlap between training and evaluation data is another important use of deduplication.
Things to know
If the threshold for 'how similar counts as a duplicate' is too loose, good documents that are actually different get deleted too; if it is too strict, duplicates remain. When data reaches billions of items, every pair of documents cannot be compared directly, so approximate techniques that compare quickly and roughly are needed. Some data is naturally repetitive (legal provisions, common code idioms), so the decision should be based on the nature of the data rather than deleting everything. Cleaning up duplicates in a document store for RAG also reduces search results being flooded with the same content.
Example
Teams that pretrain LLMs put a deduplication step in their data pipeline before documents collected from the web go into training. They filter out cases where the same news story ran on several sites and copied code files, and they also remove documents that overlap with benchmark questions so the model does not score points by memorizing them. When building RAG on internal company documents, cleaning up versions of the same document also improves search quality.