← Key papers

Paper review · November 18, 2026

InclusionMed: a medical AI benchmark and composite index for inclusive healthcare

InclusionMed: Benchmarking AI for Inclusive Healthcare

Data & evaluationLanguage modelsHealthcare & bio Beginner Upvotes 62 cs.AI

The authors built an Atlas that classifies benchmarks by 3 healthcare roles and 3 AI capabilities, plus a composite index, and compared the medical performance of 22 LLMs by role and capability.

Problem

Medical AI benchmarks are skewed toward particular roles and tasks, so it is unclear whose healthcare needs, and which ones, a single composite score reflects. A way to select and organize benchmarks that accounts for diverse users and care settings is needed.

Method

The authors built a 9-cell Atlas with two axes: healthcare role (patients and the general public, healthcare professionals, healthcare administration professionals) and AI capability (knowledge, reasoning, agency). They turned 388 candidate benchmarks into cards with LLM assistance and audited them against 6 criteria: medical relevance, reliability, availability, complementarity, timeliness and discriminative power. The remaining 8 candidates were reproduced directly to check for score saturation or weak discriminative power. Finally, they combined 4 public benchmarks (HealthBench Professional, PhysicianBench, the medical subset of ALE-CLI v1.0, CHI-Bench) and the private diagnostic reasoning benchmark InclusionMed-DxFiles, each weighted equally at 20%, into the InclusionMed Index.

Results

Limitations

Only 3 of the Atlas's 9 cells are currently filled, and there are no benchmarks for the patient and general public role or for the knowledge capability. The private benchmark cannot be reproduced externally. Because the index is an equally weighted average of 5 benchmarks, rankings may change depending on the choice of weights. Inference costs are estimates based on public pricing. CHI-Bench covers US healthcare operations cases.

Why it matters

The work gives evidence that, when adopting medical AI, organizations should look at performance and cost matched to the actual user roles and required capabilities, not a single composite score. It also reveals which healthcare needs are not yet evaluated.

Key terms

InclusionMed Atlas
A map that places benchmarks on combinations of healthcare role and AI capability (9 cells) to show which healthcare needs are evaluated and where the gaps are.
Score saturation
A state in which most models already score highly, so a benchmark can no longer distinguish differences between models.
pass@1
The rate at which a task is fully completed on a single attempt.
Pareto frontier
The set of models for which no alternative achieves a higher score at the same cost or the same score at lower cost.

Original

Authors: The InclusionMed Collaboration · Posted 2026-11-16 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.