Data-centric AI: improving models through cleaning, deduplication and synthetic data

Why it was needed: collecting more was not enough

LLMs (large language models) are pretrained on enormous amounts of text scraped from the internet. As data grew from gigabytes to terabytes, human review of each item became impossible, and quality dropped accordingly. The same advertising copy was repeated tens of thousands of times, and boilerplate such as menu and button text, along with meaningless character strings, got mixed in.

A deduplication study paper put numbers on this problem. In C4, a commonly used dataset, a single 61-word English sentence was repeated more than 60,000 times, and over 1% of the text that models trained on such data produced without any prompt was copied verbatim from the training data. All four datasets the researchers examined contained duplicates, and over 4% of the validation sets of standard datasets overlapped with the training sets.

When training and evaluation data overlap, a model looks smarter than it is. The same study paper pointed out that such overlap leads researchers to overestimate model accuracy and to choose models and settings that overfit (answer by memorizing) the training data. This is the 'data contamination' problem. No matter how much the model architecture is refined, results are hard to trust when the data looks like this.

Core idea: prepare the ingredients before cooking

Data-centric AI is an approach that improves performance by changing 'what to feed' the model while leaving the model architecture and training algorithm unchanged. In cooking terms, before changing the recipe you wash the ingredients, pick out the spoiled ones, remove some if there is too much of the same ingredient, and buy separately what is missing.

This preparation falls into three main branches. The first is cleaning and filtering, which removes texts in other languages or of low quality. The second is deduplication, which removes identical or near-identical texts. The third is synthetic data, which uses other models to create new data of the kinds that are lacking. On top of these comes data selection, which decides 'which of this data to use, and how much'.

A representative case that confirmed this idea experimentally is DataComp paper. This benchmark fixed the model training code as a standard and let participants change only how they filter the data. The design aims to compare fairly how results change when only the data changes.

How it works: the data-cleaning pipeline

The process of turning web data into training data usually follows this order. First, text is collected from public web snapshots such as Common Crawl. According to FineWeb paper, Common Crawl has been collecting the web since 2007, and many LLM pretraining datasets start from it.

Next come rule-based filters. A language identifier keeps only the desired languages, and lines that do not end with a period, lines that are too short, and pages containing banned words are discarded. FineWeb paper picked only the few that actually worked out of more than 50 candidate filters used in previous research.

The third step is deduplication. The deduplication study paper introduced two tools. One is an exact-match method that finds repeated identical strings; the other is an approximate method using MinHash, based on hashing (a technique that turns long text into a short fingerprint), to find pairs of 'near-identical' documents that share many word groups. Whether to remove at the line, paragraph or document level, and whether to remove only exact matches or similar ones too, are all design choices.

The fourth step is quality scoring. Each remaining text is scored and only high-scoring ones are kept. Scores come from perplexity (how surprising a small model finds the text) or from a classifier that judges 'is this text educational'. Synthetic data is added here if needed. Finally, a small model is trained on this data and evaluated on benchmarks, and the filter criteria are readjusted based on the results. The method FineWeb paper used to choose its filters and deduplication approach was exactly this kind of experiment (ablation) that actually trains models and compares them.

How the field developed: what the foundational studies newly showed

The deduplication study 2021 paper showed that thoroughly removing duplicates cuts how often a model emits memorized text to one-tenth, and also reduces the number of training steps needed to reach the same accuracy. Datasets became up to 19% smaller, and perplexity did not get worse and in some cases improved by up to 10%. The researchers concluded that they observed no downsides to deduplication.

In 2023, several directions emerged at once. DataComp paper built a benchmark for competing on multimodal data filtering, with a candidate pool of 12.8 billion image-text pairs and 38 evaluation sets. CLIP ViT-L/14 trained on DataComp-1B, the best baseline, reached 79.2% ImageNet zero-shot accuracy, 3.7 percentage points higher than OpenAI CLIP with the same training procedure and compute. A data pruning study paper reported that perplexity, a simple score, worked better than more expensive quality metrics, and that using only 30% of the original data could outperform the unpruned baseline.

The same year, 'Textbooks Are All You Need' paper, which introduced phi-1, showed the potential of synthetic data. Using 6 billion tokens of 'textbook-quality' data selected from the web and 1 billion tokens of synthetic textbooks and exercises generated with GPT-3.5, a 1.3-billion-parameter code model was trained for 4 days on 8 A100s. The model scored pass@1 of 50.6% on HumanEval and 55.5% on MBPP. A study on data-constrained scaling paper addressed the opposite question. When data is scarce and the same data is repeated, up to 4 epochs (passing through the data four times) shows almost no difference in loss compared with using new data, but repeating beyond that makes the gain from added compute eventually shrink to zero.

FineWeb paper in 2024 packaged this knowledge into a large public dataset. It cleaned 15 trillion tokens from 96 Common Crawl snapshots and tested and documented each design choice in filtering and deduplication. FineWeb-Edu, which selected 1.3 trillion tokens using a classifier that rates educational value, performed much better on benchmarks requiring knowledge and reasoning, such as MMLU and ARC. Its significance is that it brought data-cleaning methods, long kept hidden by companies as trade secrets, into the public domain.

Where things are heading: finer-grained scoring, cheaper selection

The first trend is to score quality against several criteria instead of a single score. Edu-QuRating paper argues that existing filters, which treat "educational value" as a single number, are too coarse, so it defines separate educational criteria such as accuracy and structure. It distills pairwise judgments, in which an LLM-as-a-Judge (an LLM used as an evaluator) compares two documents and picks one, into lightweight, reusable scoring models. The best scorer reproduced the pairwise judgments of GPT-4.1-mini with an average accuracy of 0.917. When small models were pretrained on data built by scoring 322.25 million documents, their aggregate accuracy across nine benchmarks was higher than the FineWeb-Edu baseline. However, these comparisons come from single runs.

The second trend is to lower the cost of selection. A common way to choose post-training data is to check how closely each sample's gradient (the signal that tells the model which direction to update) points in the same direction as the gradient of a small validation set. The problem is that this requires backpropagation for every sample, which is expensive. LESSER paper showed that output-layer gradients alone are enough. It cut feature-extraction compute by 9.7x for SFT and 3.0x on reinforcement learning benchmarks while keeping performance similar to using full gradients.

In short, data cleaning is moving from "discarding by rules" to "scoring with models," and then to "selecting based on how a piece of data will change the model." Data selection is being studied not only for pretraining but also for post-training stages such as fine-tuning and alignment.

Limitations and open issues

The first problem people run into is balance. FineWeb paper points out that filtering too aggressively leaves too little data to pretrain a general-purpose model. The data-constrained scaling study paper also started from the concern that web text may soon run out, and as alternatives it tested mixing in code data and dropping some commonly used filters. Quality and quantity trade off against each other.

The second is safety. A study of poisoning attacks after quality-based selection paper showed that quality filters remove many overtly harmful samples, but the "high-quality" samples that get through can still break a model's safety alignment. The researchers' Bi-QSTO method refines poisoned samples so they pass quality criteria, and more than 90% of the poisoned samples survived even a strong filter that removed 90% of the data. High quality does not mean safe.

The third is evaluation pitfalls. As the deduplication study paper showed, overlap between training and evaluation data inflates scores. Choosing filters by benchmark score also risks keeping only data that favors that benchmark. The finding that FineWeb-Edu paper did especially well on knowledge-focused benchmarks such as MMLU and ARC also shows that what counts as "good data" can depend on which benchmark you measure with.

Fourth, synthetic data carries the capability level and tendencies of the model that generated it. phi-1 paper relied on data generated with GPT-3.5, so the quality of synthetic data depends on the model that produced it and on the prompt design.

Further study

If you are new to the topic, start with the deduplication study paper. It lays out the problem, two deduplication methods and their effects briefly and clearly, which makes it a good way to get a feel for why data cleaning matters. Reading FineWeb paper next shows how a real large-scale cleaning pipeline is a series of choices and how each choice is validated through experiments.

If you are interested in quality scores, reading the data pruning study paper followed by Edu-QuRating paper shows the shift from simple scores to multi-criteria scoring. For synthetic data, the phi-1 paper paper is the starting point; for strategies when data is scarce, start with the data-constrained scaling study paper.

For multimodal data such as image-text pairs, DataComp paper shows how to design fair data experiments. If the cost of selecting post-training data is a practical concern, look at LESSER paper; if you are worried about the security of your data pipeline, also read the study of poisoning attacks after quality-based selection paper.