Paper review · October 22, 2026
When does training on synthetic data work on real data? Theoretical guarantees
When Does Training on Synthetic Data Provably Work?
The authors prove transfer error bounds for binary classifiers depending on the relationship between the synthetic and real distributions, and show through theory and experiments that there exist classifiers that are nearly perfect on synthetic data but nearly always wrong on real data.
Problem
Because high-quality training data is scarce, synthetic data produced by generative models is increasingly used. However, theoretical guarantees for when a classifier trained on synthetic data also works well on the real distribution are lacking.
Method
The authors assume the conditional distribution of labels is the same in both distributions. First, they derive finite-sample excess error bounds when the density ratio of the real input distribution relative to the synthetic distribution is controlled. Second, they analyze cases where the real distribution has regions the synthetic data does not cover (partially or fully singular). Third, for function classes where the hypotheses and log-odds are Lipschitz, they recover error bounds in terms of the probability that a real sample is not near the synthetic data. They validated the results with low-dimensional controlled experiments and with 32,200 ImageNet-counterpart images produced by 4 generative models.
Results
- The authors report showing that if the hypothesis class is expressive enough, synthetic risk alone cannot control real error, even when the Wasserstein distance is arbitrarily small.
- Under the Lipschitz condition, transfer guarantees were recovered even in the fully singular case.
- An adversarially constructed classifier recorded 0.5% error on the synthetic domain and 99.25% error on the real domain.
- For architectures trained in the usual way, synthetic test error and real test error were positively correlated.
Limitations
This analysis is based on the abstract. The results hold under assumptions such as binary classification, identical conditional label distributions between the synthetic and real distributions, and Lipschitz conditions, and the experiments are limited to low-dimensional controlled settings and one ImageNet-based image setting. Detailed experimental conditions, such as actual performance figures, should be checked in the main text.
Why it matters
As reliance on synthetic data grows, the paper shows theoretically that validation performance on synthetic data alone cannot guarantee real-world performance. The authors recommend using plenty of synthetic data but always validating on real data.
Key terms
- Absolute continuity
- A relationship in which everything that can happen under the real distribution can also happen under the synthetic distribution. In this case, the error can be transferred and computed using the density ratio of the two distributions.
- Singular distribution
- The case where some real data lies in a region that the synthetic data does not cover at all; in that region, the synthetic data provides no information.
- Wasserstein distance
- A measure of the difference between two distributions, defined as the minimum 'transport cost' of moving one distribution into the other.
Original
Authors: Leroy Odunlami, Tobias Riedlinger, Hanno Gottschalk · Posted 2026-10-20 · License arXiv 비독점 배포 라이선스 · Based on the abstract
This review is an AI-written summary of the paper. Please treat the original paper as the reference.