← Key papers

Paper review · November 19, 2026

MedJev: A System One vision-language decision model for medical imaging that decides in one pass without generation

MedJev: A System One Vision-Language Decision Model for Fast and Effective Medical Image Understanding

MultimodalComputer visionEfficiency & infraHealthcare & bio Intermediate Upvotes 10 cs.CV The Chinese University of Hong Kong

Instead of autoregressive generation, the authors built a decision model that scores candidate answers in a single forward pass, raising accuracy on medical imaging tasks by 7.93 percentage points and speed by 6.04x.

Problem

When VLMs are used to interpret medical images, they usually generate the reasoning process and the answer as tokens. As a result, longer responses cost more, and because the answer has to be parsed from free text, any response that does not match a candidate counts as wrong. Many medical tasks involve choosing one of a fixed set of candidates, so generation is not strictly necessary.

Method

Built on Qwen3.5-4B, the model places the image, the question and all candidates in a single prompt and adds question, candidate and decision marker tokens. The upper language layers, the value and FFN paths of the last layer, and the vocabulary projection were removed (about 20% of all parameters), while the vision path and all visual tokens were kept. Scores are computed with the pretrained last layer's query-key matching, using the decision marker as the query and each candidate marker as a key. A lightweight MLP residual score, with its output initialized to zero, is added on top. The model was trained on multiple medical classification and VQA datasets using weighted cross-entropy and LoRA.

Results

Limitations

It applies only to classification and multiple-choice tasks where candidate answers are given, and it cannot generate free-form answers or explanations of its reasoning. Evaluation covers four benchmarks, and latency was measured on a single H20 GPU with batch 1. How the comparison with general-purpose VLMs should be interpreted may depend on how close the training and evaluation data distributions are.

Why it matters

It shows that for medical tasks with fixed answers, such as diagnostic classification or multiple-choice reading, a decision model can be faster and more accurate than a generative VLM. Because responses always come out as valid candidates and latency is consistent, it is a form that is easy to integrate into real clinical support systems.

Key terms

System One decision model
The name comes from fast, intuitive thinking (System 1). It refers to a model that produces a score for each candidate directly from the input, without generating a reasoning process.
Query–key matching score
An approach that reuses the query and key projections of pretrained attention to score candidates by the dot product between the decision token and each candidate token.
Autoregressive generation
An approach that generates tokens one at a time in order. Each token requires a forward pass, so longer responses are slower.

Original

Authors: Fan Zhang, Hao Wu, Jinyang Wu, Hongyu Qu, Zheng Lian, Dongfu Yin, Qingsong Wen, Pheng-Ann Heng · Posted 2026-11-16 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.