Paper review · November 7, 2026
NPCBench: A clinical-training-style benchmark for multimodal care-pathway reasoning in nasopharyngeal carcinoma
NPCBench: A Clinical Apprenticeship Benchmark for Multimodal Care-Pathway Reasoning in Nasopharyngeal Carcinoma
The authors built a benchmark that structures nasopharyngeal carcinoma care step by step, from guideline knowledge to patient-specific care pathways, and showed that even models that do well on individual tasks get both the diagnosis and the treatment right for the same patient at most 23% of the time.
Problem
Existing medical benchmarks break clinical competence into pieces, testing it with isolated factual questions, single-image tasks and static cases. As a result, they do not show well whether a model integrates detailed specialist knowledge with multimodal evidence and tracks the patient's condition throughout the course of care.
Method
Modeled on specialist residency training, the benchmark has six levels (L1 guideline recall, L2 application, L3 reading pathology and MRI evidence, L4 case planning, L5 longitudinal decision revision, L6 CarePath episodes) plus an open-ended consultation rubric (O1). It contains 4,777 items and CarePaths for 100 patients (417 scored tasks). It uses the CSCO and NCCN guidelines and de-identified real patient records, and was reviewed by oncologists, pathologists and radiologists. The key metric, Strict, counts a case as a success only if the pathology, MRI T stage and all treatment answers for the same patient are correct together. Thirty models (21 multimodal) were evaluated.
Results
- Guideline recall accuracy exceeded 92%, but the best case-planning (L4) accuracy was 71.9% (Gemini-3.1-Pro-Preview).
- The best CarePath Strict success rate was 23.0% (Claude-Sonnet-4.6). Gemini-3.1-Pro-Preview reached 89.9% accuracy on treatment decisions but only 11.0% on Strict.
- The best MRI T-stage accuracy was 43.5%, slightly above the majority-class baseline (37.5%). The rate of correctly identifying involvement of all 18 anatomical structures was below 5.0%.
- Removing the answer options lowered accuracy on guideline items by 15.5 percentage points for GPT-5.1, 12.7 points for GPT-4.1 and 7.0 points for Gemini-3.1.
- Even GPT-5.1, which ranked first in consultation scores, triggered a critical-error criterion in 14.6% of its responses under the primary judge. The rubric judge showed a sensitivity–specificity imbalance compared with physician ratings.
Limitations
The benchmark covers a single disease, nasopharyngeal carcinoma, and mainly uses Chinese guidelines and data from one institution (external validation covered 53 patients). CarePath treatment decisions were selected through model-based difficulty enrichment, and some models were used in building it. With only 100 patients, the Strict figures carry large uncertainty. The consultation evaluation relies on automated judges, and judge bias was observed.
Why it matters
It broadens medical AI evaluation from per-item accuracy to the consistency of care pathways at the patient level. It shows in numbers that even with high exam-style scores, evidence reading and decisions may not line up together in a real care workflow. It has implications for how models are validated before clinical deployment.
Key terms
- CarePath episode
- A patient-level unit of evaluation that groups the linked decisions for one patient, from diagnosis through treatment and follow-up, and counts as a success only if all of them are correct.
- Composition gap
- The gap in which scores on individual tasks are high but a model fails to get multiple decisions about the same patient right at the same time.
- TNM staging
- A system that classifies how far a cancer has progressed by tumor size and invasion (T), lymph node metastasis (N) and distant metastasis (M). It serves as the basis for treatment decisions.
- Rubric-based scoring
- A method that scores open-ended answers by breaking them into items for meeting required elements and for critical and minor errors.
Original
Authors: Pengkai Wang, Wei-Wei Zhang, Yan Li, Min Tang, Zhitian Hou, Zeyu Liu, Guanghao Zhu, Yuanyi Wang, et al. · Posted 2026-11-05 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.