Paper review · November 18, 2026
Medical LLM fine-tuning that preserves safety through bilevel data selection
Safety-Preserving Medical LLM Fine-Tuning via Bilevel Data Selection
Using bilevel data selection with entropy regularization to reduce concentration of selection weights (S-BDS), the authors obtained the fine-tuning result with the highest patient-facing safety while maintaining medical performance.
Problem
SFT of a general-purpose LLM on medical data adds medical knowledge, but unsafe responses in the data can weaken the model's existing safety alignment. In HealthCareMagic-100k, only 27.68% of responses received a safety score of 1-2 (safe). Clinicians also cannot review every item in a large pool of candidate data.
Method
Per-sample selection weights are learned through bilevel optimization, with the loss on clinician-verified safety data as the upper-level objective and weighted training on candidate data as the lower-level objective (based on SEAL). To mitigate weights concentrating on a few samples and making the ranking of the rest unstable, entropy regularization is added to the upper-level objective. The extra computation is O(N). The top p% of samples are selected, combined with the safety data to form a safety-oriented dataset, and mixed with unfiltered medical task data at ratio ρ for the final SFT. Medical performance was evaluated separately with HealthBench, and patient-facing safety with PatientSafetyBench (PSB).
Results
- On Llama-3.1-8B-Instruct and Qwen3-8B-Instruct, it achieved the highest PSB Safe Rate among all compared methods, exceeding the strongest baseline by 4.64 and 3.00 percentage points respectively, while keeping a competitive HealthBench Health Score.
- In the Orca setting (Llama3.1), the top 5% of samples accounted for 42.15% of total weight under SEAL, but only 13.48% under S-BDS (λ=1). The normalized effective sample size (ESS/N) rose from 20.62% to 78.25%.
- Changing the proportion ρ of safety-oriented data in SFT made it possible to adjust the balance between medical performance and safety.
- The LLM judge used for safety assessment agreed with clinician judgments on 88.8% of a sample (Cohen's κ 0.73).
Limitations
Experiments are limited to two 8B-scale models and a single-source dataset (HealthCareMagic-100k). The safety data relies on responses revised by GPT-5.6 Sol, and safety evaluation was done mainly with an LLM judge. Medical performance was reported only as competitive, not higher than every compared method. The provided text was truncated, so the detailed comparison tables could not be checked.
Why it matters
For fine-tuning patient-facing medical chatbots, it offers a practical way to reduce safety degradation by using a small amount of verified safety data to select from a large dataset, without human review of all the data.
Key terms
- Bilevel optimization
- An optimization structure in which the upper-level objective (performance on safety data) depends on the solution of the lower-level problem (model training). Here it is used to learn data weights.
- Entropy regularization
- A regularization term that pushes a probability distribution toward a uniform distribution so it does not concentrate on one side.
- PatientSafetyBench
- A benchmark that evaluates risks from the patient's perspective, such as harmful medical advice, overconfidence, unlicensed practice of medicine, health misinformation and bias.
- Effective sample size (ESS)
- A metric of how evenly weights are spread. The larger it is, the more samples actually contribute.
Original
Authors: Ruoxuan Shi, Sheng Yang, Zhengxing Su, Xiaoyang Hou, Yating Liu · Posted 2026-11-16 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.