Paper review · October 24, 2026
From Structured Records to Patient-Facing Explanations: Preserving Clinical Fidelity in LLM Health Summaries
From Structured Records to Patient Narratives: Preserving Clinical Alignment in LLM-Generated Health Summaries
Applying evidence-constrained prompting and RAG when turning menstrual health records into patient-facing summaries improved faithfulness and the preservation of uncertainty, but reduced readability, revealing a trade-off.
Problem
When biometric records produced by wearable devices are turned into language patients can understand, LLMs can distort the meaning, a problem called semantic drift. Prior work on medical text simplification rewrites reference documents in simpler terms, so it has not addressed checking faithfulness against the source data itself (structured observations).
Method
The authors built 500 semi-synthetic menstrual health records, using a causal DAG to capture dependencies among cycle, sleep, heart rate, HRV and symptoms. They also added missing data and confidence scores by user type. With GPT-4o, Qwen2.5-14B and Phi-4-mini, they generated 4,500 summaries under three settings: Base, constrained prompting (Constrained), and constraints plus RAG over README lay-term definitions (RAG-Constrained). Summaries were evaluated on five categories: faithfulness violations, epistemic mismatch (stating uncertain content as certain), diagnostic overreach, omissions, and urgency mismatch. Evaluation used PubMedBERT NLI, scispaCy and an LLM judge (Claude-3.5-Haiku), and readability and jargon density were also measured.
Results
- The average faithfulness violation rate fell from 0.242 (Base) to 0.188 (Constrained) and 0.178 (RAG-Constrained). GPT-4o dropped from 0.186 to 0.152, but Phi-4-mini rose slightly from 0.212 to 0.220.
- Epistemic mismatch decreased monotonically for all three models (average 0.297 → 0.263 → 0.248). The number of uncertainty-related claims actually increased (1.17 → 1.76 → 2.70).
- Excluding explanatory mentions related to pregnancy, diagnostic overreach was close to 0 in all conditions (maximum 0.006 for Phi-4-mini).
- The effect on omissions varied by model. GPT-4o improved from 0.219 (Base) to 0.130.
- The authors report that as clinical fidelity improved, reading difficulty and the use of medical terms increased.
Limitations
The data are 500 semi-synthetic single-day observations rather than real patient records, and are limited to the single domain of menstrual health. Evaluation relies on automatic metrics such as an NLI model, keyword-based rules and an LLM judge, and the authors themselves note the domain limitations of these tools. There is no evaluation by real patients or clinicians. Constraints were applied only at the prompt level.
Why it matters
It shows with numbers that when healthcare apps use LLMs to produce patient-facing explanations, faithfulness, expression of uncertainty and readability need to be considered together. A practical takeaway is that prompt constraints and RAG alone can reduce overly certain statements, but this conflicts with writing in plain language.
Key terms
- Semantic drift
- The gradual distortion or loss of the original meaning when specialized medical content is rewritten in language for the general public.
- NLI (natural language inference)
- The task of deciding whether a premise sentence entails a hypothesis sentence, is neutral toward it, or contradicts it. It is used to check whether generated text is factually grounded.
- Atomic fact decomposition
- A method that breaks a summary into claims that cannot be split further, then checks one by one whether each claim is supported by the evidence.
- RAG
- A method that retrieves external knowledge (here, lay definitions of medical terms) before generation and inserts it into the prompt.
Original
Authors: Bayram Ayadi, Florian Geissler, Johanna Schmidhuber · Posted 2026-10-22 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.