← Key papers

Paper review · November 12, 2026

dNaViT: Discrete codes as general-purpose visual representations

dNaViT: Discrete Codes are General-Purpose Visual Representations

MultimodalComputer visionGenerative media Advanced Upvotes 32 cs.CV LongCat

By adding Residual Modulation and Grouped Product Quantization to a frozen vision encoder, the authors built a visual tokenizer that handles understanding, generation, editing and reconstruction with a single discrete vocabulary.

Problem

Language models are unified around discrete tokens and next-token prediction. Multimodal models, by contrast, use separate semantic features for understanding and reconstruction latents for generation. Converting visual signals into a finite discrete vocabulary creates an information bottleneck. It is also hard to fit both high-level semantics and pixel detail through a single bottleneck.

Method

A ResMod (Residual Modulator) is placed on top of a frozen, pretrained native-resolution vision encoder. ResMod fuses intermediate features from multiple depths and adds scale and shift residuals to the final semantic features. It is initialized to zero, so the original features are preserved at the start. The channels are then split into G groups and quantized with a separate codebook per group (Grouped Product Quantization). This increases representational capacity without lengthening the spatial sequence. Training uses a dual-branch loss for semantic feature reconstruction and pixel reconstruction, and proceeds in three stages: continuous initialization → discrete feature alignment → joint optimization.

Results

Limitations

Within the available text, the scale of the training data and the detailed conditions for each compared model are hard to verify fully. The authors note that increasing the number of groups G increases the number of code indices to predict, adding to the burden of generative modeling. Large code capacity does not guarantee reconstruction quality. Generality beyond the combination of the backbone encoder and the language model (Qwen3-8B) is outside the scope of validation.

Why it matters

Treating images as discrete tokens like language allows understanding and generation to be unified in a single next-token prediction model. Language-model training infrastructure and methods can then be reused as they are. This makes it an important component for teams designing native multimodal models.

Key terms

Visual tokenizer
A module that converts an image into a sequence of codebook indices and reconstructs the image from them. It lets a language model handle images like text tokens.
Grouped Product Quantization
A method that splits a feature vector into several groups and quantizes each group with its own codebook. With K codebook entries × G groups, K^G combinations can be represented.
Residual Modulation
A method that adds per-position scale and shift corrections to frozen features, injecting the detail needed for reconstruction while preserving the original semantic structure.

Original

Authors: Xiaotong Li, Zhengrong Yue, Xu Huang, Jialv Zou, Minhao Jing, Wenhao Yan, Haodong Li, Hai Huang, et al. · Posted 2026-11-09 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.