Diffusion and Flow Matching: Two Paths from Noise to Images

Why it was needed: the limits of earlier generative models

Image generation models came in several families before diffusion. DDPM paper notes in its introduction that GANs (generative adversarial networks), autoregressive models, VAEs and normalizing flows had produced convincing images and audio. But as the latent diffusion paper paper pointed out, GANs involve complex adversarial training and do not scale well to diverse distributions, so their successes were concentrated on data with relatively little variation, and they suffered from mode collapse (repeatedly producing only one or two kinds of output) and training instability. Autoregressive transformers had to be scaled to billions of parameters to handle high-resolution scenes.

Diffusion models emerged in this gap. They are simple to define and efficient to train, but before DDPM there had been no demonstration that they could produce high-quality images. DDPM showed that these models can in fact produce samples comparable to, and sometimes better than, other generative models.

A new limit soon appeared, however. According to the latent diffusion paper, training the most powerful diffusion models took 150–1000 V100 GPU-days, and because the same network had to be run 25–1000 times in sequence, generating 50,000 images on a single A100 took about 5 days. Much of the later research answers the question of 'how to reduce this cost while keeping quality'. Flow matching is also an answer to that question, from a somewhat different angle.

Core idea: running spreading ink in reverse

A useful analogy for diffusion is ink dropped into clear water. Over time the ink spreads evenly until its original shape can no longer be made out. This 'spreading process' follows fixed rules, so anyone can imitate it. A diffusion model learns the reverse: starting from the fully spread state, it steps backward one step at a time to recover the shape of the original ink drop. In terms of images, the process of gradually mixing Gaussian noise into a photo until it becomes pure noise is the 'forward' process, and the process of gradually removing noise from that noise to produce a photo is the 'reverse' process. Only the reverse process is learned.

Flow matching describes the same destination in a different way. It assumes there is a path along which a point in noise space moves to a point in data space, and it learns a map of arrows that tells, at every point on that path, 'which direction to move and how fast right now' — that is, a velocity field (vector field). With this map, you only need to start at the starting point and move little by little along the arrows to arrive at the data. This 'following the arrows' computation is solving an ordinary differential equation (ODE), and off-the-shelf ODE solvers can be used as is.

The two views are closer to an inclusion relationship than to competitors. The flow matching paper paper states that the family of Gaussian probability paths it covers includes existing diffusion paths as a special case. In other words, diffusion is 'one of many possible paths', and flow matching lets you choose the path itself. The key motivation is that by choosing a straighter path (an optimal transport path), you can reach the same destination in fewer steps.

How it works: the steps of training and generation

Training runs in the following order. First, pick one clean image from the training data. Second, sample a random time step t. Third, mix in the amount of noise corresponding to that time step. An important property of the forward process noted by DDPM paper is that a noisy image at any time step can be produced in one shot with a closed-form expression, without mixing in noise t times in sequence from the start. This makes it possible to sample time steps at random and train efficiently with stochastic gradient descent. Fourth, give the neural network (the denoiser) the noisy image and the time step and have it predict what should be removed. DDPM showed that with a particular parameterization, this training becomes equivalent to denoising score matching over multiple noise levels, and the generation process comes to resemble Langevin dynamics (a method that moves gradually toward higher probability while mixing in noise); this approach gave the best quality.

Training in flow matching paper has a similar structure. It samples a point on a predefined conditional probability path between noise and data, and has the neural network predict the target velocity at that point by regression. The paper calls this 'simulation-free' training, because the correct velocity can be computed directly from a single point on the path, without actually solving the ODE and following the Trajectory during training.

Generation goes in reverse. Starting from pure Gaussian noise, it repeatedly calls the trained network and moves one step at a time toward the data. With diffusion, each step removes a little noise; with flow matching, each step moves along the velocity field. Since each step is one pass through a large neural network, the number of steps is the cost of generation.

Real systems add two more pieces. One is the latent space. Latent diffusion paper first uses an autoencoder to compress images into a lower-dimensional space that is perceptually equivalent, runs diffusion in that space, and finally restores the image with a decoder. The other is conditioning input. The same paper added cross-attention layers so that generation can take conditions such as text or bounding boxes. The denoiser's backbone was initially a U-Net, and DiT paper replaced it with a transformer operating on latent patches.

How the field developed: what each of the four foundational papers added

DDPM 2020 paper was the first to clearly show that diffusion models can produce high-quality images. It recorded an Inception score of 9.46 and an FID of 3.17 on unconditional CIFAR10, and achieved sample quality similar to ProgressiveGAN on 256×256 LSUN. It also cited the equivalence between diffusion and denoising score matching/Langevin dynamics as a main contribution. At the same time, it stated its own limitations. Its log-likelihood (a measure of how plausibly a model explains the data) fell short of other likelihood-based models, and most of the lossless code length was spent describing imperceptible fine details.

Latent diffusion 2021 paper targeted exactly that 'compute wasted on imperceptible details'. The paper views learning as two stages, perceptual compression (removing high-frequency detail) and semantic compression (learning content and composition), assigning the former to an autoencoder and letting diffusion focus on the latter. The autoencoder is trained only once and can be reused across multiple diffusion models. By using cross-attention to accept general conditions such as text, it established the basic framework for text-to-image generation, and it achieved strong performance on tasks such as inpainting while greatly reducing compute requirements compared with pixel-based diffusion.

Flow matching 2022 paper redesigned the training objective itself. It made it possible to train continuous normalizing flows (CNFs, models that turn a simple distribution into the data distribution through continuous transformations) at scale without simulation, by regressing the vector field of a fixed conditional probability path. Using flow matching with diffusion paths allowed diffusion models to be trained more stably, and paths defined by optimal transport (OT) displacement interpolation were more efficient than diffusion paths, giving faster training and generation and better generalization. It reported consistently better likelihood and sample quality on ImageNet than diffusion-based alternatives.

DiT 2022 paper changed the backbone. It replaced latent diffusion's U-Net with a transformer that treats latent patches like tokens, and demonstrated scalability: FID decreased steadily as compute (Gflops) grew through increasing depth and width or the number of input tokens. The largest model, DiT-XL/2, recorded an FID of 2.27 on class-conditional ImageNet 256×256. This provided evidence that scaling-law thinking — 'make the transformer bigger and it gets better' — also applies to image generation.

Where things are heading: faster, toward what you want, and deeper

The first direction is speed. SoftCache paper starts from the observation that in diffusion and flow models, the pass through a large neural network at every step dominates generation cost. Existing training-free caching methods reused recent computation results (activations) and recomputed when a cheap signal crossed a threshold, an 'on/off' approach. This work notes that how far the signal is from the threshold (the margin) predicts reuse error, and proposes 'soft caching', which skips several fixed steps when the margin is large and shortens the stride as a fresh computation approaches. It reported that on FLUX for images and HunyuanVideo and Wan2.1 for video, it outperformed the baselines on all four fidelity metrics even in the 3.5–6.9× speedup range. That it requires no additional training matters in practice.

The second direction is fine-tuning models toward desired outcomes. Weighted Adjoint Matching paper is a method for fine-tuning pretrained diffusion and flow matching models when a differentiable reward is available. Building on the original Adjoint Matching, which framed fine-tuning as a stochastic optimal control problem with a quadratic control cost, it uses the observation that these models produce coarse features first and details later. Early in generation it sets the control cost high to preserve the diversity of meaning-bearing coarse features, and near the end it lowers the cost to refine details and raise the reward. The paper proved that this is equivalent to maximizing reward under time-varying KL regularization, and reported a better trade-off between reward and mode balance in Gaussian mixture and MNIST experiments. The small scale of the experiments should be taken into account.

The third direction is the theoretical question of whether 'more steps means deeper thinking'. A self-conditioning study paper points out that, in the line of work that solves visual reasoning problems by generating images, the intuition that each step looks like one step of reasoning may be wrong. If the output satisfies a few mild conditions, the Trajectory of a basic flow matching sampler can be simulated with a constant number of parallel evaluation rounds, so adding steps does not increase the expressive power of a fixed-depth backbone model. In contrast, self-conditioning — feeding the final image estimated at the previous step as input to the next step — removes this bottleneck, and in experiments accuracy and reasoning depth increased, with performance improving as the number of steps grew. The paper proposes the design principle 'pass more information between steps'.

Limitations and open issues

The most fundamental limitation is still the cost of iterative generation. Latent spaces, straighter paths and caching have all reduced this cost, but the structure itself — 'passing through a large neural network many times' — remains. Because training-free acceleration such as SoftCache paper skips steps outside the fixed step grid, it must be continually checked for which models and settings quality holds up.

The second is the gap between metrics and real quality. DDPM paper stated that its sample quality is good but its log-likelihood is not competitive, citing as the reason that most of the code length is spent on imperceptible details. Numbers such as FID are useful for comparing studies, but they are values for a specific dataset and resolution and should not be read directly as real-world service quality.

The third is the tug-of-war between control and diversity. Fine-tuning with a reward improves results in the desired direction but can skew outputs to one side. This is why Weighted Adjoint Matching paper measured 'mode balance' separately. What to change, and by how much, at each time step becomes the key design variable.

The fourth is caution about the intuition that 'iteration = reasoning'. As the self-conditioning study paper showed, simply increasing the number of steps does not make a model perform deeper computation. Using generative models as reasoning tools means designing the structure of what is passed between steps, which also connects to the broader question of how test-time compute should be used.

Further study

If you are new to the topic, start with DDPM paper. The introduction and Section 2 alone cover the forward and reverse processes, the closed-form expression that produces a noisy image at any time step in one shot, and how the training objective simplifies. If the math is heavy going, focus on Figure 2 (the graphical model) and the 'connection to denoising autoencoders' part of Section 3.

Next is latent diffusion paper. Figure 2 in the introduction (perceptual and semantic compression) explains 'why latent space' most intuitively. The basic skeleton of today's text-to-image systems (autoencoder + diffusion + cross-attention conditioning) comes from here.

Then move on to flow matching paper to learn the 'choose the path' view and the advantages of optimal transport paths, and read DiT paper for why the backbone switched to transformers and its scalability analysis. After these four papers, among recent work, read SoftCache paper if your interest is practical, Weighted Adjoint Matching paper if it is fine-tuning, and the self-conditioning study paper if it is theory.