Paper review · October 31, 2026
Statistical analysis of verified synthetic data: A linear regression framework
Statistical Analysis of Synthetic Data with Verification: A Linear Regression Framework
Formalizing synthetic data generation, verification and fine-tuning in linear regression, the authors derive finite-sample risk bounds and show that stricter verification is not always better.
Problem
Repeated self-training on synthetic data can amplify a pretrained model's errors and cause model collapse. Selecting only good samples through verification is widely used, but how the verification threshold, generation noise and fine-tuning method affect prediction risk has not been well established theoretically.
Method
After building an OLS pretrained estimator from n real samples, synthetic responses are created by adding generation noise (scale σ̃) to m downstream task covariates. An oracle verifier accepts only samples whose error relative to the true regression value is at most δ, and the accepted samples are used to re-estimate all coordinates (full fine-tuning) or some coordinates (partial fine-tuning). The authors derive non-asymptotic risk bounds in four regimes defined by the relative sizes of δ, σ̃ and the pretraining error scale √(d/n), and extend the analysis to iterative verified fine-tuning.
Results
- In partial fine-tuning, risk follows a U shape as the verification threshold shrinks, first decreasing and then increasing again, so stricter verification did not always lower risk.
- Moderate generation noise reduced prediction risk, but too much noise increased risk again.
- Full and partial fine-tuning responded differently to the same verification rule, because the pretraining error in the fixed coordinates affects both sample selection and the resulting risk.
- For iterative verified full fine-tuning, the authors give upper and lower bounds showing that the contribution of the initial error decays exponentially when the threshold is sufficiently smaller than the generation noise.
Limitations
The analysis assumes linear regression with Gaussian covariates, an OLS estimator and an oracle verifier that knows the true error exactly, which is far from the reality of LLM fine-tuning. Most results are rates that hold only up to constant factors, and the scope of real-data experiments could not be verified in the provided text.
Why it matters
It gives a theoretical account of the trade-offs to consider when setting thresholds, sampling temperature and whether to use partial fine-tuning such as LoRA in synthetic data filtering or rejection-sampling-based fine-tuning.
Key terms
- Model collapse
- A phenomenon in which a model's performance gradually degrades as errors accumulate from repeatedly training on data it generated itself.
- Verification threshold δ
- A criterion that accepts a synthetic response for training only when its error is at most this value. Smaller values are stricter but also reduce the number of accepted samples.
- Partial fine-tuning
- An approach that updates only some parameters and keeps the rest fixed at their pretrained values. It is a simplified setting of parameter-efficient fine-tuning.
Original
Authors: Heejune Sheen, Harrison H. Zhou · Posted 2026-10-29 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.