Paper review · November 19, 2026
MedJev: A System One vision-language decision model for medical imaging that decides in one pass without generation
MedJev: A System One Vision-Language Decision Model for Fast and Effective Medical Image Understanding
Instead of autoregressive generation, the authors built a decision model that scores candidate answers in a single forward pass, raising accuracy on medical imaging tasks by 7.93 percentage points and speed by 6.04x.
Problem
When VLMs are used to interpret medical images, they usually generate the reasoning process and the answer as tokens. As a result, longer responses cost more, and because the answer has to be parsed from free text, any response that does not match a candidate counts as wrong. Many medical tasks involve choosing one of a fixed set of candidates, so generation is not strictly necessary.
Method
Built on Qwen3.5-4B, the model places the image, the question and all candidates in a single prompt and adds question, candidate and decision marker tokens. The upper language layers, the value and FFN paths of the last layer, and the vocabulary projection were removed (about 20% of all parameters), while the vision path and all visual tokens were kept. Scores are computed with the pretrained last layer's query-key matching, using the decision marker as the query and each candidate marker as a key. A lightweight MLP residual score, with its output initialized to zero, is added on top. The model was trained on multiple medical classification and VQA datasets using weighted cross-entropy and LoRA.
Results
- Average accuracy across four benchmarks — dermoscopy images (HAM10000), chest X-rays (PneumoniaMNIST), multi-image cases (MedThinkVQA) and medical paper figures (PMC-VQA) — rose from 61.13% for the original Qwen3.5-4B to 69.06% (+7.93 percentage points). Accuracy improved on all four benchmarks
- Average end-to-end latency fell from 685.03 ms to 113.44 ms (6.04x, single H20, batch 1). The speedup was largest on the multi-image benchmark, which has the longest responses
- Compared with 20 general-purpose and medical VLMs in the 4B–9B range, the authors report it was the only model to rank first in both average accuracy and average speed
Limitations
It applies only to classification and multiple-choice tasks where candidate answers are given, and it cannot generate free-form answers or explanations of its reasoning. Evaluation covers four benchmarks, and latency was measured on a single H20 GPU with batch 1. How the comparison with general-purpose VLMs should be interpreted may depend on how close the training and evaluation data distributions are.
Why it matters
It shows that for medical tasks with fixed answers, such as diagnostic classification or multiple-choice reading, a decision model can be faster and more accurate than a generative VLM. Because responses always come out as valid candidates and latency is consistent, it is a form that is easy to integrate into real clinical support systems.
Key terms
- System One decision model
- The name comes from fast, intuitive thinking (System 1). It refers to a model that produces a score for each candidate directly from the input, without generating a reasoning process.
- Query–key matching score
- An approach that reuses the query and key projections of pretrained attention to score candidates by the dot product between the decision token and each candidate token.
- Autoregressive generation
- An approach that generates tokens one at a time in order. Each token requires a forward pass, so longer responses are slower.
Original
Authors: Fan Zhang, Hao Wu, Jinyang Wu, Hongyu Qu, Zheng Lian, Dongfu Yin, Qingsong Wen, Pheng-Ann Heng · Posted 2026-11-16 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.