ALoDLM: A Looped Diffusion Language Model That Adjusts Iteration Depth per Token

Problem

Diffusion language models (DLMs) are fast because they predict multiple tokens in parallel. But their quality is lower than autoregressive (AR) models of the same size. The authors attribute this to a 'compute-difficulty mismatch': within one denoising step, easy and hard tokens get the same depth of computation. Confidence-based decoding also throws away the latent computation for deferred tokens and recomputes them from scratch in the next step.

Method

The Transformer is split into three parts, Prelude, Recurrent Core and Coda, with an inner loop inside each denoising step. Tokens whose prediction entropy falls below a threshold are committed, and their embeddings are fed back into the context. Uncommitted tokens keep their latent states and pass through the Recurrent Core repeatedly. A separate ExitGate head decides when to stop the inner loop. In training, the per-token exit depth (exit schedule) is treated as a latent variable and a conditional NELBO is derived, jointly optimizing token prediction and compute allocation. The discrete exit decisions are learned with an unbiased score-function gradient estimator and variance reduction techniques. Qwen3 1.7B and 8B backbones were directly SFT-trained on a 5-billion-token corpus.

Results

• The average score across 11 reasoning, math and coding benchmarks was 65.5 at 1.7B and 80.3 at 8B. The authors report that these are higher than all compared DLMs and the corresponding Qwen3 AR baselines.
• In parallel decoding on GSM8K, ALoDLM-8B reportedly had about 2.7x higher throughput than Qwen3-8B served with vLLM, with similar or higher accuracy.
• The authors' analysis suggests iteration depth can serve as an axis for test-time scaling. They also report that the model learns different numbers of refinement steps per token without explicit difficulty labels.

Limitations

Experiments are limited to the 1.7B and 8B scales, with Qwen3 backbones SFT-trained on 5 billion tokens. The effect when pretraining from scratch or at larger scales has not been verified. Speed comparisons focus on GSM8K, so the quality-efficiency trade-off on other tasks needs separate checking. Because exit decisions are learned with a score-function estimator, extra machinery such as variance reduction is required.

Why it matters

The work narrows the DLM quality gap through token-level adaptive computation without reverting to an AR architecture. Because it showed both a higher average score than an AR model and faster serving at the 8B scale, it is a useful reference for judging whether diffusion-based LLMs are practical to use.

Key terms

Diffusion language model (DLM)

A model that generates text by restoring a masked sentence over several steps. It is fast because it can predict multiple tokens in parallel in a single step.

Looped (recurrent-depth) transformer

An architecture that applies the same block of layers repeatedly to increase computational depth without adding parameters.

NELBO

The negative evidence lower bound. It is an upper-bound objective minimized in place of the log-likelihood, which is hard to compute directly. Here it was used to treat the exit schedule as a latent variable.

Score-function estimator

A method (REINFORCE) for estimating gradients when there is a non-differentiable choice, such as discrete sampling, by multiplying the gradient of the log probability by the resulting cost.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Liancheng Fang, Zhuowei Li, Youngeun Kim, Tianchen Zhao, Rajat Koner, Jiaye Wu, Linghan Xu, Xuanbai Chen, et al. · Posted 2026-10-03 · License CC BY-NC-SA 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.