Medical AI: from image reading to clinical LLMs

Why medicine needed AI

Medicine is a field where a single judgment directly affects a patient's body. Take detecting pneumonia on chest X-rays as an example. The CheXNet paper paper, citing statistics from the US Centers for Disease Control and Prevention (CDC), states that in the US alone more than 1 million adults are hospitalized with pneumonia each year and about 50,000 die from it. Chest X-rays are the best available method for diagnosing pneumonia, but they still require a radiologist to read them. That is the problem the paper starts from.

Reading the images is not easy either. According to the same paper, pneumonia appears faint on X-rays and can overlap with other conditions or resemble harmless abnormalities. As a result, even radiologists often disagree. The paper also cites as motivation that automated reading could help patients in areas with limited access to radiologists.

The problem on the language side is different in nature. A study of LLMs and clinical knowledge paper pointed out that medical AI up to that point consisted of single-task systems for classification, regression or segmentation, which did not fit the real clinical workflow of asking and answering questions in words. This led to hopes of using LLMs (large language models) for knowledge retrieval, clinical decision support, summarizing key findings and initial patient sorting (triage). The same paper also named as risks hallucination, where a model makes up plausible medical misinformation, and bias that could widen health disparities.

Core idea: reader and advisor, exam and clinic

Medical AI is easier to understand if you divide it into two broad branches. The first is image-reading models. They take a single image and answer one fixed question, such as 'what is the probability that this image shows pneumonia?' Their scope is narrow, but they go deep within it. The second is clinical LLMs. They are closer to an advisor that takes a question in text and answers in text. They can cover a broad range, but it is harder to predict where they will go wrong.

Both branches face the same question: 'Does doing well on an exam mean doing well in the clinic?' It is similar to how a perfect score on a written driving test does not guarantee real driving skill on the road. Exam questions have set answers and present information cleanly. In real clinical practice, information may be missing, irrelevant details may be mixed in, or a doctor may simply accept the AI's answer as is.

So the field's development can be viewed along two axes. One is how well models solve problems. The other is the evaluation methods and regulation that examine whether those scores can be trusted as real clinical performance. This article covers the two axes in turn.

How it works

CheXNet paper is a 121-layer convolutional neural network (CNN, a neural network that reads local patterns in an image layer by layer). Given a frontal chest X-ray downscaled to 224×224, it outputs the probability of pneumonia along with a heatmap that colors the regions behind its judgment. Starting from weights pretrained on ImageNet, a general photo dataset, it was trained on ChestX-ray14, a dataset of 112,120 images from 30,805 patients. The disease labels in this dataset were automatically extracted from radiology reports.

Two features of this process are worth noting. One is how it handled class imbalance, the fact that pneumonia cases make up a small share of the data: errors were weighted differently according to the ratio of positives to negatives. The other is that when splitting the training, validation and test sets, the same patient was never placed in two sets. If images of the same patient appear in both training and testing, scores come out better than they really are.

Clinical LLMs work differently. The clinical knowledge study paper used Flan-PaLM, which adds instruction-following training to PaLM, a 540-billion-parameter model. It had the model solve multiple-choice questions using a mix of few-shot prompting (showing a few examples), CoT (Chain-of-Thought, prompting step-by-step reasoning) and self-consistency (having the model answer several times and choosing the most frequent answer). It then applied instruction prompt tuning, which adapts the model to the medical domain with a small number of examples, to create Med-PaLM. The follow-up, Med-PaLM 2 paper, switched the base model to PaLM 2 and added medical-domain fine-tuning and ensemble refinement prompting, which gathers several answers and refines them.

In both cases, the model's output is information that clinicians consult when making decisions. The GMLP principles document source, covered in the regulation section below, also stresses looking at the performance of the human-AI team rather than the model alone.

How the field developed: from reading scores to clinician panel evaluation

CheXNet paper achieved an F1 score of 0.435 on a test set of 420 images read by four practicing radiologists. The radiologists' average was 0.387. The F1 score is a metric that reflects both missed patients and wrongly flagged patients. When the same model was extended to 14 chest diseases, it also exceeded all the best published results at the time. However, this comparison was measured on 420 images from a single dataset, using a single metric, F1.

The clinical knowledge study paper broadened the evaluation. It assembled MultiMedQA, a benchmark spanning medical licensing exams, medical research and consumer questions, and proposed a framework in which humans rate answers along several axes, such as factuality, precision, possibility of harm and bias. Flan-PaLM scored 67.6% on MedQA, which consists of US Medical Licensing Examination (USMLE)-style questions, beating the previous best by more than 17%. But when a clinician panel evaluated long-form answers, only 61.9% agreed with scientific consensus. For Med-PaLM, this figure was 92.6%, similar to clinician answers (92.9%), and the share of answers that could lead to harmful outcomes fell from 29.7% for Flan-PaLM to 5.8% for Med-PaLM, similar to clinicians (6.5%). Even so, the authors concluded that Med-PaLM still fell short of clinicians.

The GPT-4 medical evaluation paper reported that a general-purpose model without medical-specific training exceeded the USMLE passing score by more than 20 points, without any special prompting. The study also measured how accurately the model estimates the likelihood that its answers are correct (calibration), and GPT-4 was much better calibrated than GPT-3.5. This is an important property in a high-stakes field like medicine. The study also examined whether the model might have memorized exam content during training.

Med-PaLM 2 paper scored up to 86.5% on MedQA. When physicians compared pairs of answers side by side for 1066 consumer medical questions, they preferred Med-PaLM 2's answers over physician-written answers on 8 of 9 axes related to clinical utility. On 240 'adversarial' questions designed to probe the model's limits, it also improved over Med-PaLM on every evaluation axis. Still, the authors clearly stated that further research is needed to confirm its effectiveness in real clinical settings.

Where things are heading: finding failures outside the exam

Recent research is shifting away from pushing scores higher and toward measuring the failures that occur under real clinical conditions. A study on vulnerability to incidental information paper examined ambient documentation, where clinical conversations are recorded automatically. Across 576 patient-physician conversations, current models included small talk unrelated to care in 35% of notes. In 3.7% of notes, they attributed that small talk to the wrong person or used it as if it were clinical information. In 57 recorded simulated visits, background speech from another patient's visit got into 48.2% of transcripts, and that contamination appeared in 5.3% of the follow-up notes produced by four open-weight models. The researchers proposed that robustness to this kind of incidental information should be evaluated before clinical use.

MedicalHarness paper pointed out that a medical agent's score comes not from a single model but from 'the combination of a model and a harness (the system that controls the iterative loop between the model and its environment)'. When five open-weight models were run with five harnesses on 107 tasks across four domains, the harness and the harness-model interaction explained about one quarter of the variation in results. No single harness was best for every model and task. This means the same model can get different benchmark scores depending on the framework it is placed in.

The 'Decide, Ask, Defer' study paper looked at how models behave when information is insufficient. It separately evaluated whether a model diagnoses right away (DECIDE), asks for more information (ASK) or hands off to clinicians (DEFER). When Qwen, Gemini and GPT were placed in the same 200 clinical information states, the models chose different actions in 137 of the 200 states. Offering three options prevented some incorrect autonomous decisions, but it also eliminated some decisions that would have been correct. The researchers reported plainly that the policy was not an improvement on every dimension.

Regulation and governance: the 10 GMLP principles and WHO recommendations

The US FDA, Health Canada and the UK Medicines and Healthcare products Regulatory Agency (MHRA) jointly set out 10 Good Machine Learning Practice (GMLP) principles for developing medical devices that use machine learning source. Starting with the data-related principles: clinical study participants and datasets should represent the intended patient population in terms of age, sex, race, ethnicity and similar factors (Principle 3), and training and test data should be independent of each other, taking into account all dependency factors such as patient, data acquisition method and site (Principle 4). The reference standard should be built using the best available methods, and its limitations should be understood (Principle 5).

There are also principles on evaluation and operation. Where the setup involves people in decisions, that is, Human-in-the-Loop, the document says to focus on the performance of the human-AI team rather than the model alone (Principle 7), and to test under clinically relevant conditions and in important subgroups (Principle 8). Users should be given clear information on the intended use, performance by subgroup, characteristics of the training data and known limitations (Principle 9). Deployed models should be monitored for real-world performance, and when retrained, risks of performance degradation such as overfitting, unintended bias and data drift should be managed (Principle 10).

On January 18, 2024, WHO released guidance on the ethics and governance of large multimodal models (LMMs) for health source. The guidance contains more than 40 recommendations aimed at governments, technology companies and healthcare providers. It identifies five areas of use: diagnosis and clinical care, patient-led symptom investigation, administrative tasks such as summarizing electronic health records, medical and nursing education, and research and drug development. It recommends that governments designate a regulatory agency to assess and approve LMMs for health use, and mandate post-release audits and impact assessments by independent third parties when LMMs are deployed at scale. Audit results should be published and broken down by user type, such as age, race or disability. It recommends that developers involve clinicians and patients from the early stages of development and design models to perform well-defined tasks with the necessary accuracy and reliability.

Limitations and issues: bias, accountability and patient safety

The first issue is bias. The WHO guidance source warns that LMMs may be trained on data that is biased or of poor quality with respect to race, ethnicity, ancestry, sex, gender identity or age. This is also why the GMLP principles source call for representative data and disclosure of performance by subgroup. Even when a model's average score is high, its performance can be low for particular patient groups. The reference standard has limitations too. The ChestX-ray14 labels that CheXNet paper was trained on were automatically extracted from radiology reports, which connects to GMLP Principle 5 on understanding the limitations of the reference standard.

The second is accountability and automation bias. WHO source lists as a risk 'automation bias', in which clinicians and patients trust an LMM too much, missing errors they would otherwise have caught or inappropriately handing difficult decisions to the model. However high a test score is, the final judgment on diagnosis and treatment, and the responsibility for it, rests with clinicians. GMLP Principle 7's call to look at human-AI team performance and the 'Decide, Ask, Defer' study paper separately evaluating the option of handing off to clinicians follow the same line of thinking.

The third is over-reading scores. MedQA 86.5% or 'answers preferred over physicians'' come from fixed questions and evaluation conditions. These results should not be read as real-world clinical performance. The authors of Med-PaLM 2 paper also wrote that further research is needed to confirm effectiveness in real clinical care. As the incidental information study paper and MedicalHarness paper show, results change when only the noise in the exam room or the system configuration changes.

The fourth is security and privacy. WHO source noted that LMMs are vulnerable to cybersecurity risks, which can threaten patient information and the trustworthiness of algorithms. GMLP Principle 2 also requires, as a baseline, security practices that protect the authenticity and integrity of data.

Further reading

To see how an image-reading model is built and evaluated, CheXNet paper is a good place to start. It is a short paper, but it covers dataset construction, patient-level splitting, handling class imbalance and comparison with specialists all at once. The confidence intervals in Table 1 also give a sense of the uncertainty in a comparison on the scale of 420 images.

For clinical LLMs, we recommend reading the clinical knowledge study paper first. It shows most clearly how multiple-choice scores and clinician panel evaluations can paint different pictures. Then read Med-PaLM 2 paper and the GPT-4 medical evaluation paper and compare score gains, calibration and the possibility of memorization.

If you need a practical and regulatory perspective, read the 10 GMLP principles source like a checklist, and check the recommendations for governments and developers in the WHO release source. For recent evaluation trends, reading the incidental information study paper, MedicalHarness paper and the 'Decide, Ask, Defer' study paper in that order gives a sense of how 'failures beyond the test' are measured.