Paper review · October 31, 2026
LOWN: Diffusion VLM decoding that looks at the image only when needed
Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs
By skipping image processing when the image-free prediction agrees with the most recent image-conditioned prediction, it retained 99.6% of LLaDA-V's performance at 35.3% of the decoding FLOPs.
Problem
Diffusion vision-language models (DVLMs) reprocess the entire input at every denoising step, and 89–97% of the input is visual tokens. Visual token pruning loses information while still processing the remaining image at every step. The authors measured that the tokens being unmasked changed in only 22.8% of steps on average when the image was removed.
Method
The top-1 predictions and confidences from the last FULL (image-included) forward pass are stored as the anchor. At each step, a cheap TEXT-ONLY forward pass is run, and the longest prefix, in order of confidence, that agrees with the anchor predictions is "certified". If the number of certified tokens is at least the default unmasking count k, all certified tokens are unmasked without a FULL forward pass. Otherwise, a FULL forward pass is run to update the anchor, and k tokens are unmasked starting from the position where the two predictions differ. It requires no training or threshold tuning and can be combined with pruning and KV caching.
Results
- On LLaDA-V, it retained 99.6% of the baseline's average score across 10 benchmarks at 0.353× the decoding FLOPs.
- In FLOPs-matched comparisons, it outperformed RedVTP (93.0%), ZOO-Prune (94.2%) and KLASS (91.9%), and achieved the top score on 8 of 10 benchmarks.
- Combined with KV caching or visual token pruning, it further reduced those methods' FLOPs by 1.8–2.5×, and together with KV caching it decoded 5.6–7.0× faster than the base model.
- The share of steps that needed the image varied by benchmark, from 7.9% on MMBench to 43.8% on DocVQA, showing that adaptive decisions are needed rather than a fixed schedule.
- The same rule was also applied to LaViDa and the video model VidLaDA.
Limitations
Of the tokens unmasked in TEXT-ONLY steps, 2.2–4.7% would be flipped by a new FULL forward pass, so approximation errors remain. Scores dropped on some benchmarks (MathVista 59.10→56.00, MME 1719.49→1655.08). Efficiency was reported mainly in analytical FLOPs, and the default experiments use batch size 1 and greedy decoding. Target models are limited to LLaDA-family DVLMs.
Why it matters
The idea of doing expensive computation only when the image is actually needed can substantially reduce the inference cost of diffusion multimodal models without training. It can also be combined with existing pruning and caching.
Key terms
- Diffusion VLM (DVLM)
- A vision-language model that, instead of producing a response one token at a time from left to right, fills in a masked response in parallel over multiple denoising steps.
- Visual token pruning
- A technique that reduces computation by removing some of the image tokens the model processes. The removed information is lost.
- Anchor and certificate
- LOWN treats tokens on which the last image-included prediction (the anchor) and the current text-only prediction agree as "trustworthy without the image" and reuses them.
Original
Authors: Donghyeon Cho, Joowon Kim, Seungho Shin, Eunho Yang · Posted 2026-10-27 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.