Training on synthetic data: from Self-Instruct to the model collapse debate

Why it was needed: the limits of human-written data

An LLM that has only gone through pretraining is good at predicting the next word but poor at following human instructions. So it goes through instruction tuning (SFT, supervised fine-tuning), training again on pairs of instructions and model answers. The problem is that people have to write this data by hand. Self-Instruct paper pointed out that human-written instruction data is limited in quantity, lacks diversity and creativity, and is concentrated on common NLP tasks, limiting the generality of tuned models.

Difficulty is another problem. WizardLM paper found that human-written instructions skew toward easy and medium-difficulty problems, with few difficult ones. The explanation is that human writers tire easily, making it hard to consistently produce many high-difficulty instructions. The high cost and time of labeling are also a major burden.

This led to the idea of "letting the model create the data too." Humans provide only a small number of seeds, and the LLM generates instructions and answers in bulk. Data made this way is called synthetic data. A survey of synthetic data paper cites the difficulty of obtaining large, diverse, high-quality data due to data scarcity, privacy concerns and high costs as the reason synthetic data is drawing attention.

Core idea: a student who writes and solves their own problem set

Here is an analogy. A teacher shows a student only a few example problems. The student imitates those examples to write new problems and solutions, and throws away any that are nonsense or nearly identical to existing ones. The remaining problems are collected into a workbook, and the student studies from that workbook. In effect, this draws out, in the form of problems, abilities the student already had but could not put to use.

The analogy also carries a risk. What happens if the student studies only from workbooks they made, and the student trained that way then makes the next workbook, over and over? Rare problem types the student did not know well gradually drop out of the workbooks, and only familiar types remain. It is like copying a copy: each time, the details of the original get blurrier. This is the intuition behind model collapse, covered later.

In the end, training on synthetic data comes down to two key questions. One is how to create and filter good data. The other is how to avoid losing the diversity of the real world as synthetic data piles up.

How it works: generate → filter → train

The pipeline of Self-Instruct paper is the basic framework. It starts by filling a task pool with 175 human-written tasks (one instruction and one example per task). It then goes through four steps. First, it samples a few tasks from the pool, shows them in a prompt, and has a language model generate new instructions. Second, it determines whether each new instruction is a classification task. Third, it generates input-output examples for the instruction. Here, depending on the task type, it uses either an input-first approach or an output-first approach (generating the answer label first). Fourth, it filters out low-quality instructions and those too similar to existing ones using several rules (heuristics).

Tasks that survive go back into the task pool and are used as examples for the next round of generation. This loop makes the data snowball, and once enough has accumulated, the data is used to fine-tune the very model that generated it. The paper describes this process as an alignment method that requires almost no labeling.

Later methods change individual steps of this framework. In the generation step, they make instructions harder (Evol-Instruct) or have the model write textbook-style text (phi-1); in the filtering step, they add more sophisticated selection criteria. The generating model and the model being trained can also be different. For example, training a small model on data generated by a larger model is effectively close to Distillation (knowledge distillation).

How the field developed: more data, harder data, more textbook-like data

Self-Instruct paper obtained about 52K instructions and about 82K input-output examples from vanilla GPT3. Tuning GPT3 on this data improved it by 33 percentage points (absolute) over the original model on the Super-NaturalInstructions benchmark, bringing it roughly on par with InstructGPT-001, which was trained on private user data and human labels. In a human evaluation using new instructions written by experts, the gap with InstructGPT-001 was 5 percentage points. The generated tasks overlapped little with the seed tasks and departed from the typical distribution of NLP tasks, achieving the original goal of diversity to some extent.

Evol-Instruct from WizardLM paper added difficulty as another axis alongside diversity. It has an LLM rewrite a single instruction: in the depth direction, it uses five operations (adding constraints, deepening, concretizing, increasing reasoning steps, and complicating the input); in the breadth direction, it uses a mutation that creates an entirely new instruction from the given one. Instructions that fail to evolve are filtered out in an elimination step. Starting from 52K Alpaca examples built with the Self-Instruct method, the authors evolved them four times with ChatGPT to obtain 250K instructions. To compare fairly with Vicuna, which was trained on 70K real user conversations, they sampled 70K of these to train LLaMA 13B. In automatic evaluation by GPT-4, they reported that the model reached more than 90% of ChatGPT's capability on 17 of 29 skills.

phi-1 paper put synthetic data at the center of the training data, not just post-training. It combined 6B tokens of "textbook-quality" data selected from the web with 1B tokens of synthetic textbooks and exercises generated by GPT-3.5, and trained a 1.3B-parameter code model on 8 A100s for 4 days. The result was 50.6% pass@1 on HumanEval and 55.5% on MBPP, and a 350M model trained the same way (phi-1-small) also reached 45% on HumanEval. It is often cited as evidence that data quality and format can matter more than quantity.

A survey of synthetic data paper gathers these results while stressing that synthetic data must ensure factuality, fidelity to the original, and unbiasedness, and must be used responsibly. What happens when the conditions this survey points out are not met is the subject of the next debate.

The model collapse debate: replace or accumulate

The model collapse debate began with Curse of Recursion paper. This work asks what the next generation of models will grow up on once LLMs write a large share of the text on the internet. It argued that training on model-generated content makes the tails of the original data distribution, that is, rarely occurring cases, disappear, and that this defect is irreversible. It named this phenomenon model collapse and showed that it can occur in VAEs (variational autoencoders), Gaussian mixture models, and LLMs alike. It therefore concluded that data left behind by real human interactions will become more valuable.

The rebuttal started from the experimental assumptions. Is Model Collapse Inevitable? paper pointed out that earlier studies mostly assumed new data replaces old data, but that it is more realistic to assume data accumulates. When the authors pretrained language models over several generations, replacing real data with synthetic data each generation did lead toward collapse, but continuing to accumulate each generation's synthetic data alongside the original real data did not cause collapse. This result held across changes in model size, architecture, and hyperparameters, and was similar for diffusion models for molecular structure generation and VAEs for images. Using a simple linear model theory, they also proved that with replacement, test error grows with each iteration, whereas with accumulation it has a finite upper bound independent of the number of iterations.

A recent study paper notes that the two conclusions appear to contradict each other and examines a realistic condition between them in diffusion models. In this condition, both past data and real data are kept, but because the training budget is fixed, each new model is trained on a fixed-size sample drawn from a growing data pool. Even though no data is deleted, the proportion of real data gradually approaches zero. In experiments on 2D spiral data, MNIST, Fashion-MNIST, and CIFAR-10, the replacement approach degraded quickly as in earlier studies, while the fixed-budget approach degraded only partially. Some features disappeared within a few generations under both approaches, but others were preserved practically indefinitely under the fixed-budget approach. This means collapse is not all-or-nothing and can happen differently for each feature.

Where things are heading: what to select

Since the collapse debate, the practical question is less "Can we use synthetic data?" and more "Which synthetic data, and how much?" The TATC study paper starts from the observation that more synthetic data does not always make a model better. Useful synthetic data must add information needed for the target task without bringing in errors that eat away at that gain. Also, the same example can have different value as the training set grows. Using linear theory, this study analyzed when synthetic data is useful, how much to add, and the marginal value of adding a single example. Based on this, it proposed TATC (Training-Aware Target Coverage), a selection method that picks, among candidates that help the target task, examples that expand directions the existing data does not yet cover. When fine-tuning Qwen2.5-Math-1.5B-Instruct for math reasoning, it outperformed other selection methods on GSM8K across several selection budgets.

A study on data groups paper goes a step further. Current data curation usually estimates the training utility of each sample separately, but the study argues that for synthetic data this alone is not enough, and group-level signals that reflect interactions between samples are needed. Synthetic datasets with different compositions can be indistinguishable by individual-level criteria yet differ greatly by group-level criteria, and selecting by group-level criteria gave better performance, especially on generative capabilities. As the share of synthetic data in the training pipeline grew, only those widely used curation methods that reflected group signals beat the baseline.

The two studies point in the same direction: they turn the filtering that Self-Instruct did with simple similarity heuristics into a selection problem that considers the target task and the composition of the data as a whole. Whereas the collapse studies focused on the proportion of real data and how it is kept, this line of work focuses on which combinations to keep even within synthetic data.

Limitations and open issues

First, there are doubts about evaluation methods. The "more than 90% of ChatGPT's capability" reported by WizardLM paper is the result of automatic evaluation by GPT-4. Because a model trained on LLM-generated data is scored again by an LLM, its ability may be overestimated if the evaluator prefers a similar style or format. LLM-as-a-Judge results should be viewed together with human evaluations or benchmarks with ground-truth answers.

Second, the conclusions of collapse studies depend heavily on experimental conditions. Which of replacement, accumulation, or fixed budget is closer to real-world web crawling or in-house data pipelines differs from case to case. The optimistic result of the accumulation experiments paper relies on the premise that real data is never discarded, and the fixed-budget experiments paper show that some features can be lost even under that premise. Therefore, neither "collapse does not happen" nor "collapse always happens" can be accepted as a general conclusion.

Third, there is the problem of guaranteeing quality. As the synthetic data survey paper stresses, synthetic data can contain factual errors and biases. If a generating model's hallucinations become fixed in training data, they can carry over to the next model. Also, cases like phi-1 paper, where a small model scored high, came from a narrow domain such as code where answers can be verified, so it is hard to conclude that the same will hold in other domains.

Fourth, there is the paradox that real data becomes more valuable. As Curse of Recursion paper concludes, data from real human interactions is becoming scarcer, and identifying and keeping "real" data on an internet mixed with synthetic data becomes a data governance task in itself.

Further study

If you are new to the topic, start with Section 2 (Method) of Self-Instruct paper. Its four steps (seed tasks, instruction generation, input-first/output-first generation, and filtering) form the backbone of almost all later research on synthetic instruction data. Next, the Evol-Instruct figure in WizardLM paper gives a sense of how "making instructions harder" is implemented through prompt operations. phi-1 paper is the representative case of how much data quality and format can make up for model size.

For model collapse, read Curse of Recursion paper and Is Model Collapse Inevitable? paper as a pair. After confirming that the two papers reached different conclusions because of whether data was replaced or accumulated, moving on to the study of the fixed-budget condition paper gives a clear view of the whole debate.

To apply this in practice, get the big picture from the synthetic data survey paper, then read TATC paper on selection methods and the group-level curation study paper to set criteria for deciding what to keep.