Paper review · November 18, 2026
InclusionMed: a medical AI benchmark and composite index for inclusive healthcare
InclusionMed: Benchmarking AI for Inclusive Healthcare
The authors built an Atlas that classifies benchmarks by 3 healthcare roles and 3 AI capabilities, plus a composite index, and compared the medical performance of 22 LLMs by role and capability.
Problem
Medical AI benchmarks are skewed toward particular roles and tasks, so it is unclear whose healthcare needs, and which ones, a single composite score reflects. A way to select and organize benchmarks that accounts for diverse users and care settings is needed.
Method
The authors built a 9-cell Atlas with two axes: healthcare role (patients and the general public, healthcare professionals, healthcare administration professionals) and AI capability (knowledge, reasoning, agency). They turned 388 candidate benchmarks into cards with LLM assistance and audited them against 6 criteria: medical relevance, reliability, availability, complementarity, timeliness and discriminative power. The remaining 8 candidates were reproduced directly to check for score saturation or weak discriminative power. Finally, they combined 4 public benchmarks (HealthBench Professional, PhysicianBench, the medical subset of ALE-CLI v1.0, CHI-Bench) and the private diagnostic reasoning benchmark InclusionMed-DxFiles, each weighted equally at 20%, into the InclusionMed Index.
Results
- Of the 388 candidates, 4 public benchmarks remained. Three knowledge benchmarks were excluded for score saturation, and HealthBench for weak discriminative power.
- Index scores for the 22 models ranged from 17.5 to 49.1 out of 100. The top models were Gemini 4 Argon (49.1), Claude Opus 5.5 (48.9) and Claude Fable 5.1 (48.9).
- Rankings differed by role and capability. For example, GPT-6 Luna ranked 8th in reasoning and 20th in agency. Scores for the healthcare administration role were consistently lower than for the healthcare professional role.
- On HealthBench Professional, all of the top 10 models scored higher on patient communication than on clinical decision-making. Among models with similar Index scores, average inference cost per task differed by about 24x in some cases.
Limitations
Only 3 of the Atlas's 9 cells are currently filled, and there are no benchmarks for the patient and general public role or for the knowledge capability. The private benchmark cannot be reproduced externally. Because the index is an equally weighted average of 5 benchmarks, rankings may change depending on the choice of weights. Inference costs are estimates based on public pricing. CHI-Bench covers US healthcare operations cases.
Why it matters
The work gives evidence that, when adopting medical AI, organizations should look at performance and cost matched to the actual user roles and required capabilities, not a single composite score. It also reveals which healthcare needs are not yet evaluated.
Key terms
- InclusionMed Atlas
- A map that places benchmarks on combinations of healthcare role and AI capability (9 cells) to show which healthcare needs are evaluated and where the gaps are.
- Score saturation
- A state in which most models already score highly, so a benchmark can no longer distinguish differences between models.
- pass@1
- The rate at which a task is fully completed on a single attempt.
- Pareto frontier
- The set of models for which no alternative achieves a higher score at the same cost or the same score at lower cost.
Original
Authors: The InclusionMed Collaboration · Posted 2026-11-16 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.