Paper review · October 13, 2026

DiffusionART: Efficient, Stable and Scalable Reinforcement Learning for Diffusion Models

DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models

Generative media Reinforcement learning Advanced Upvotes 228 cs.LG

By aggregating per-prompt rewards into a single improvement direction and adding an update that applies restoration in proportion to drift, the method makes reinforcement learning for diffusion models efficient and stable without EMA tuning.

Problem

Online reinforcement learning for fine-tuning diffusion models with black-box rewards comes in two families. Reverse-process methods such as Flow-GRPO have imprecise credit assignment, which makes them inefficient. Forward-process methods such as DiffusionNFT are efficient, but training is unstable without Off-policy training through an EMA tuned for each task.

Method

The authors first show that NFT-style objectives contain a hidden 'forward-reverse consistency force'. When the positive and negative branches are trained together, this force cancels out only partly. They also show that EMA stabilizes training not because of Off-policy Rollouts but because of a restoration effect that pulls the model back toward a delayed reference point. Building on this, DiffusionART adds two things. (1) It aggregates per-prompt samples and rewards into a shared reward-improvement direction G(c), and regresses the velocity at states visited in reverse-process Rollouts toward a target along this direction, which removes the consistency force by design. (2) It replaces EMA with a second update, masked by a trust region, that applies restoration proportional to per-state drift. It does not need a differentiable reward model and also supports pairwise-comparison reward models (DiffusionART-Pairwise).

Results

On HunyuanVideo-13B, with visual quality (VQ) and motion quality (MQ) rewards weighted 1:1, it reached the late-stage average reward of DiffusionNFT and Flow-GRPO-Fast with 12.02× and 11.72× fewer GPU hours, respectively.
In the same setting, the authors report an average training reward 28.8% higher than DiffusionNFT and 53.0% higher than Flow-GRPO-Fast.
In image-to-video generation, it needed 4.8× fewer GPU hours than Flow-GRPO-Fast to reach an MQ of 6.
On SD3.5-M, it matched the efficiency of the strongest baseline without EMA or task-specific tuning.
In an ablation, updating NFT twice per Rollout without EMA avoided collapse but made training 15.2× slower. The authors present this as evidence that restoration is the key stabilizing factor.

Limitations

The main figures are based on average reward during training, so quality gains measured by independent evaluation metrics are hard to confirm from the excerpted text. The analysis covers rectified flow models that use linear interpolation. The authors leave a detailed analysis of why restoration comes to dominate gradient aggregation to future work.

Why it matters

For reinforcement learning of image and video generation models on human preference or quality rewards, it reduces per-task EMA tuning and cuts GPU cost substantially, which makes it practical for post-training large video models.

Key terms

Flow-GRPO / DiffusionNFT: These are the two families of reinforcement learning for diffusion models. Flow-GRPO is a reverse-process method that optimizes denoising transition probabilities, while DiffusionNFT is a forward-process method that adds noise back to generated outputs and performs reward-weighted regression.
EMA (exponential moving average) Rollout policy: A technique that generates samples with an averaged model that slowly tracks the parameters of the model being trained. This paper argues that EMA's stabilizing effect comes from a restoring force that pulls the model back toward this averaged model.
Credit Assignment: The problem of splitting a reward given only for the final result into contributions from intermediate steps (here, each denoising step).
Trust-region: A mechanism that limits how much a single update can change the policy, so it does not move too far from the previous policy.

Original

Authors: Zhenglin Wan, Jiayu Chen, Zhengfeiyang Zhang, Pengfei Zhou, Wangbo Zhao, Fangyuan Kong, Pengfei Wan, Kun Gai, et al. · Posted 2026-10-08 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.