Paper review · October 28, 2026
MedVTok: A Universal Medical Visual Tokenizer
MedVTok: A General-Purpose Medical Visual Tokenizer
A tokenizer that maps 2D images and 3D volumes across 9 modalities into a single token space, trained with slice differential regularization and multi-expert alignment, reported state-of-the-art performance on 24 benchmarks.
Problem
Existing medical visual tokenizers are tied to a specific dimensionality (2D or 3D) and imaging modality, so multimodal medical AI has to handle incompatible representations separately. Building a single tokenizer requires preserving anatomical continuity between slices of a 3D volume while also capturing the clinical meaning that differs by modality.
Method
Inputs are split into patches regardless of dimensionality, and a Transformer encoder-decoder with coordinate-based RoPE processes 2D and 3D together. In addition to reconstruction losses (pixel, LPIPS, adversarial), it adds Slice Differential Regularization (SliDR), which makes the first- and second-order differences between adjacent slices match the original. For semantic learning it uses Multi-Expert Representation Alignment (MERA): features from modality-specific expert encoders are projected into a shared space for contrastive learning (shared), and the tokenizer representations are mapped back into each expert space to match cosine similarity (expert). It is trained on about 28.76 million images and about 70,000 volumes, with 2D only in stage 1 and alternating 2D and 3D in stage 2.
Results
- In 2D reconstruction (9 modalities), it ranked first on both PSNR and SSIM for every modality, with an average PSNR of 40.18dB and SSIM of 94.16%, higher than the runner-up AToken (38.22dB, 91.72%).
- The gap was largest for pathology images: a PSNR of 30.16dB, higher than AToken (28.57dB) and MedITok (26.03dB).
- In 3D reconstruction it scored highest, with CT PSNR of 36.50dB (AToken 35.68) and MRI PSNR of 31.77dB (AToken 31.46).
- The authors report state-of-the-art performance on 24 benchmarks covering classification, retrieval, segmentation, synthesis, VQA and radiology report generation.
Limitations
The provided text cut off after the reconstruction experiments, so the numbers and baseline comparisons for downstream tasks such as classification, segmentation and VQA could not be checked. There is far less 3D data (70,000 volumes) than 2D data (28.76 million images), and the gap with AToken in 3D reconstruction is small. All evaluations freeze the tokenizer and assess only token quality; no clinical validation is included.
Why it matters
If medical multimodal models can take various images such as CT, MRI, pathology and fundus images as tokens in the same format, the burden of attaching a separate encoder for each modality decreases, and it becomes easier to handle multiple tasks with one model. It can be seen as a candidate input interface for general-purpose medical foundation models.
Key terms
- Visual tokenizer
- An encoder-decoder that converts images or volumes into compressed token sequences that models can handle easily, and can reconstruct them back.
- Slice Differential Regularization
- A loss that makes the changes between adjacent slices in a 3D volume (first- and second-order differences) match the original. It reduces discontinuities between slices.
- Multi-Expert Representation Alignment
- A method that aligns tokenizer representations with the features of multiple modality-specific expert encoders. It uses shared-space alignment and expert-space alignment together to preserve modality-specific information.
- PSNR / SSIM
- Metrics that measure how closely a reconstructed image matches the original. PSNR is based on pixel error and SSIM on structural similarity; higher is better for both.
Original
Authors: Chenglong Ma, Yuanfeng Ji, Junzhi Ning, Jiyao Liu, Yujie Wei, Shaohao Rui, Chaoyang Zhang, Wenjie Li, et al. · Posted 2026-10-26 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.