Deep dives · 2026-11-17 · Advanced · By bombong
Drawing in a few steps: step distillation for diffusion models
Diffusion models are slow because they remove noise over dozens of repeated steps. Step distillation transfers a teacher model that takes many steps into a student model that draws in 1–4 steps. This article focuses on distribution matching (DMD) and covers how it works, what it trades off, and recent trends.
Contents
- Step distillation splits into trajectory-preserving methods, which learn to follow the teacher's 'drawing path', and distribution matching methods, which match the 'distribution of outputs'. The former is stronger on diversity, the latter on sharpness.
- DMD corrects the student using, as its signal, the difference between the teacher's prediction and the prediction of a fake score model that tracks the student's distribution. Fitting this fake model accurately decides whether it succeeds or fails.
- In text-to-image generation, an analysis found that the term coming from CFG, rather than distribution matching itself, actually drives distillation. This started a trend of designing each part of DMD separately.
- The biggest open issue is that diversity collapses as the number of steps drops. Proposed fixes include splitting objectives by noise range and restoring internal activations.
- Recent work is moving toward optimizing reinforcement learning rewards and distillation together, or targeting even more extreme step counts such as 1–2 steps or 1.x steps.
Why it was needed: the cost of an image drawn by repeated erasing
A diffusion model starts from pure noise and completes an image by removing noise little by little. Each removal step requires running a huge neural network once in full. The number of these network calls is called NFE (Number of Function Evaluations), and getting a high-quality image usually takes dozens. 1.x-Distill paper used the original model's '28×2 NFE' sampling as its baseline. This means taking 28 steps and calling the network twice at each step.
The reason for calling it twice is CFG (Classifier-Free Guidance). CFG computes a prediction with the prompt and one without it, then amplifies the difference between them so the model follows the prompt more faithfully. It helps quality but doubles the computation. A single image costs dozens of calls, and video costs more in proportion to the number of frames. This is heavy for real-time services or on-device generation.
That is why step distillation (knowledge distillation), which transfers the ability of a model that draws well in many steps (the teacher model) to a model that draws in only a few steps (the student model), became a core stage of post-training for diffusion models. The goal is simple: cut NFE to a single digit, ideally to 1–2, while preserving quality and diversity.
Core idea: learn the path, or learn the destination?
Step distillation has two main branches. Think of it as the path down from a mountain peak to a village. The first is the trajectory-preserving approach. It remembers the exact path the teacher took down and teaches the student to jump across shortcuts on that path in one go. Consistency distillation belongs to this family. The second is the distribution matching approach. It ignores the path and teaches the student so that the photos of villages it arrives at form a distribution that is 'broadly similar' to the photos the teacher arrives at. The representative method is DMD (Distribution Matching Distillation).
The two approaches trade off different things. CrossDistill paper summarized this tension as follows: trajectory-based methods cover the data's multiple modes (different types of outcomes) evenly and preserve diversity, while distribution matching makes results sharper but can reduce diversity. Recent work such as Trajectory-Regularized Distribution Distillation (TRD) also points out that trajectory alignment performs worse at very low step counts. SwiftVideo paper likewise found that using only one approach leads to performance collapse or more artifacts at low step counts.
As a result, most actual research has developed by putting distribution matching at the center and filling in its weaknesses (diversity collapse, unstable training) with other mechanisms. Below, we first look at how distribution matching works.
Two branches of step distillation
Trajectory preservationLearn the path the teacher took
- Covers multiple modes evenly, keeping diversity
- Performance drops at very few steps
- Consistency distillation family
Distribution matching (DMD)Match the distribution of final outputs
- Sharper results
- Diversity may drop
- Fake score model must track the student
공통 Both transfer a many-step teacher into a few-step student
How it works: the three models of DMD
In DMD, three models work together. Following the description in eDMD paper, step by step: First, the student generator produces an image from noise in a few steps. Second, noise is deliberately mixed back into this image. Third, the noisy image is shown to two models. One is the frozen teacher model, which indicates 'if this were real data, the noise should be removed in this direction'. The other is the fake score model, which keeps training on images the student makes and indicates 'for the student's distribution, this direction'. Fourth, the difference between the two predictions becomes the training signal that pulls the student toward the teacher.
The advantage of this design is that there is no need to generate large amounts of results in advance by actually running the teacher for dozens of steps. The student gets its signal directly on its own outputs. There is a price, though: the fake score model must accurately track the constantly changing student distribution. VA-TDM paper pointed to exactly this as a weakness. At the intermediate steps of few-step sampling, the score of the student distribution is hard to estimate, and commonly used estimators can describe a distribution different from the one that actually updates the student.
CFG adds to this. Decoupled DMD paper decomposed DMD's training objective and reported that in tasks that need CFG, such as text-to-image generation, it is not distribution matching but the 'CFG Augmentation' term that is the engine actually driving distillation. The distribution matching term acts as a regularizer that stabilizes training and reduces artifacts. The study found that this regularizing role is not unique to distribution matching either: it can be replaced with simpler non-parametric constraints or GAN-based objectives, though each has different pros and cons.
How it developed: several directions for reworking DMD
The studies covered here take DMD as a starting point and fix different weaknesses. An early example of the hybrid direction is SwiftVideo paper on the video side. It preserves trajectories with continuous-time consistency distillation while applying 'dual-perspective alignment', which combines distribution alignment between synthetic and real data with trajectory alignment across different inference steps, and it reported better results than existing methods on the OpenVid-1M benchmark. CrossDistill paper split this combination along the noise axis. Based on the observation that early high-noise steps set the overall composition (mode) and later low-noise steps refine details, it assigned a trajectory-preserving objective to the early range and a distribution matching objective to the later range.
The second direction is combining with reinforcement learning. DMDR paper changed the practice of doing distillation and reinforcement learning (the stage that aligns the model with human preferences) separately in sequence, and optimized them together instead. Reinforcement learning enables preference-aware distillation rather than compressing the whole data distribution equally, while DMD acts as a regularizer that prevents reward hacking (where only the reward score rises while actual quality drops) during reinforcement learning. DMDR reported top-level quality among few-step generation methods and said it even surpassed the multi-step teacher model. RTDMD paper showed that an objective matching a 'reward-tilted teacher distribution' naturally splits into a distribution matching term and a reward maximization term, and reported top-level preference, aesthetics and composition scores with only 4 steps on SD3, SD3.5 and FLUX.2.
The third direction is extreme step counts. 1.x-Distill paper found that DMD suffers diversity collapse and loss of fidelity at 2 steps or fewer, and corrected the role the teacher's CFG plays in mode collapse. It then used a two-stage strategy: first learning large-scale structure with diversity-preserving distribution matching, then refining details with adversarial distillation. By also building block-level caching into training, it reached an effective NFE of 1.67 on SD3-Medium and 1.74 on SD3.5-Large, and reported being up to 33× faster than the original 28×2 NFE sampling. It opened up a non-integer '1.x-step' regime.
The fourth direction is theoretical unification. VA-TDM paper proposed Velocity Adversarial Distillation, in which a learned critic model compares the velocity fields (the direction in which noise is removed) of the student and teacher and turns the mismatch into a training signal. It argued that this framework unifies score distillation, Fisher-based matching and moment matching, and reported top performance at 4 steps for both text-to-image and text-to-video.
How far step counts were cut
Where things are heading now
Studies released in November 2026 diagnose in finer detail 'exactly what goes wrong' in DMD. eDMD paper questioned the practice in multi-step distillation of training the fake model on intermediate states or across all steps. Only the final output reaches the user, but because the training signal gets entangled with intermediate distributions, the signal may not go to zero even when the final output already matches the teacher. So it uses a single fake model trained only on the student's final outputs for corrections at every step, and showed theoretically that with this setup the distribution matching term vanishes only when the final output distribution matches the teacher.
The ReDiverse study paper looked for the cause of diversity collapse inside the model. A distilled student keeps the total variance of its activations close to the teacher's, but concentrates that variance in far fewer directions. This directional collapse was concentrated in specific middle blocks at the first denoising step. In other words, the model had not lost generative ability; it had misallocated variance. Based on this, the study proposed a training-based method that aligns student activations toward the teacher with a lightweight adapter (ReDiverse-FT) and a training-free method that pushes activations toward the teacher's principal component directions at inference (ReDiverse-TF), and reported restoring diversity while preserving quality for both images and video.
NitroFusion2 paper changed what the discriminator looks at in adversarial distillation. Noting that in transformers that process images and text together (MMDiT), the intermediate states of text tokens have already 'read' the image, it used these as the discriminator's input. As a result, it reduced few-step distillation, commonly 4–8 steps, to 1–2 steps at similar quality, and reported sharp text rendering and detail even at 1 step. In short, the current trend is redesigning DMD's components one by one: 'what to train the fake model on', 'where diversity is lost', and 'where to get the discriminator signal'.
Limitations, open issues and practical criteria
The biggest issue is diversity collapse. The ReDiverse study paper described the phenomenon where results look nearly the same even with different random seeds as a 'severe' problem, and 1.x-Distill paper found the problem stands out at 2 steps or fewer. If a service samples several candidates from the same prompt and picks one, it should not adopt a distilled model based only on benchmark quality scores. Check directly whether changing the seed actually changes the results.
The second is the reliability of the training signal. VA-TDM paper and eDMD paper both point out that if the fake score model fails to track the student properly, training goes in the wrong direction. RTDMD paper also added a consistency regularizer so the fake score model tracks the moving student distribution within a limited number of updates. If training is unstable or artifacts appear, suspect the fake model before the student.
The third is reward hacking when combined with reinforcement learning. DMDR paper found DMD can be used as a regularizer against reward hacking, but this design presupposes the risk that reward optimization drifts away from the teacher distribution. It should be paired with human review that checks whether only preference scores are rising or the actual outputs are improving.
To summarize the criteria: if real-time performance or cost matters and single-image quality is key, around 4 steps with a distribution matching focus is a reasonable starting point. If diversity matters, also consider hybrids that mix in trajectory preservation or diversity-correction techniques. 1–2 steps are reaching practical levels in research, but check separately the items that matter for your own service, such as text rendering or detail, before using them. A common mistake is looking only at the fact that it is faster than the teacher without separating the effect that relies on CFG from the effect of distribution matching. As the Decoupled DMD paper analysis shows, the two terms do different jobs.
Checks before adopting a distilled model
Quality and diversity
- Do results actually change when the seed changes?
- Have key items such as text and detail been checked separately?
- Do artifacts stay flat as step count drops?
Training
- Does the fake score model track the student well?
- Have CFG effects been separated from distribution matching effects?
Combining with reinforcement learning
- Is there reward hacking where only the reward score rises?
- Do people review the outputs directly?
Further reading
To grasp the principles first, start with Decoupled DMD paper. It splits DMD's training objective into an engine (CFG augmentation) and a shield (distribution matching), which makes it easier to understand why later studies fix those parts. Next, the introduction of eDMD paper briefly summarizes the relationship between the teacher, fake and student models, which makes it a good review of how the structure works.
If you are interested in the diversity–quality trade-off, read CrossDistill paper and the ReDiverse study paper together. The former addresses the same problem from the external axis of noise ranges, the latter from activations inside the model. If you are interested in extreme step counts, compare 1.x-Distill paper and NitroFusion2 paper.
To handle preference alignment at the same time, read DMDR paper and then RTDMD paper. For applying these to video generation, SwiftVideo paper and VA-TDM paper show cases of extending image techniques to video.
Related terms
Diffusion model Distillation GAN Reinforcement learning Reward hacking Post-training Inference Transformer
References
Foundational papers
- Decoupled DMD: CFG Augmentation as the Spear, Distribution Matching as the Shield (2025)
- 1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation (2026)
- CrossDistill: Balancing Quality and Diversity via Trajectory-Level Hybrid Few-Step Distillation (2026)
- SwiftVideo: A Unified Framework for Few-Step Video Generation through Trajectory-Distribution Alignment (2025)
- Distribution Matching Distillation Meets Reinforcement Learning (2025)
- Reinforcing Few-step Generators via Reward-Tilted Distribution Matching (2026)
- VA-TDM: A Unified View for Improving Few-Step Diffusion Models by Velocity Adversarial Distillation (2026)
Recent research
Written by AI from the listed references and reviewed. Please check the original sources for details.