Paper review · October 19, 2026
MindEval2: Evaluating Mental Health Counseling LLMs with Clinically Grounded Simulated Patients and Rubrics
MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support
The authors evaluated 18 LLMs by combining a rubric designed by clinical psychologists with a patient simulator that reproduces multi-session counseling. They found that guidance calibration is the main source of failure for all models.
Problem
More people are using AI chatbots for mental health support, but evaluation methods are lacking. Simulated patients in existing benchmarks are overly cooperative and self-reflective. LLM-as-a-judge scores also rely on Likert scales that are hard to interpret. And they did not evaluate the longitudinal aspect of counseling the same patient over several sessions.
Method
The framework has two components. ① MHQR: together with PhD-level psychologists, the authors organized 14 failure types in counselor responses into 6 categories (reinforcing distortions, factual and contextual fidelity, guidance calibration, boundaries, delivery, and other). Each turn is given a severity (no issue/Minor/Major/Critical, weighted 0/1/5/10), and these are averaged into a score. The design draws on MQM, a machine translation evaluation method. ② MindSim: builds LLM patients from clinically grounded patient profiles. At each turn, the patient first decides on an 'action' (disclosing, avoiding, pushing back, etc.) through a tool call and then speaks. Between sessions, a shared memory of up to 30 entries is updated to reproduce multi-session counseling.
Results
- On 66 turns jointly annotated by 3 clinical psychologists, severity agreement (quadratic-weighted κ) was 0.44. Exact agreement was about 49%, but 97.5% differed by at most one level.
- All 5 candidate Judge models exactly reproduced the ranking of 3 counseling models based on the annotators' combined judgments (Kendall τb=1). Bootstrap ranking stability was 87% for GPT-6.1 Sol and 89% for GPT-6 Astra, close to the annotators' own stability (90%).
- GPT-6.1 Sol came within 0.01 in severity macro F1 of the best Judge, Claude Opus 5.5, at about 1/10 the cost per query. Hy4-preview and DeepSeek-V4.1-Flash caught fewer issues and were judged unsuitable as Judges.
- The authors report that in the evaluation of 18 models, recent models have greatly improved on reinforcing distortions and boundary issues. In contrast, the guidance calibration problem, repeating exploratory questions without giving concrete guidance, remained in all models and accounted for most of the quality gap.
- In multi-session counseling, models that used content from earlier sessions scored better. Models that did not tended to ask again about things the patient had already said.
Limitations
The severity weights (1/5/10) are only an aggregation rule, not values calibrated to actual ratios of clinical harm. Judge validation used only 411 turns and 3 counseling models. The ranking comparison also covers just three pairs, so its scope is narrow. Inter-annotator severity agreement is moderate, and agreement on individual issue types is lower. The evaluation is limited to everyday, non-crisis counseling and uses simulated patients. The provided text was cut off partway, so detailed figures for the 18 models could not be checked.
Why it matters
Mental health chatbots are a field where a single response can cause real harm to a user. Because this framework marks issue types and severity per turn, developers can see specifically what to fix. It also points to a direction beyond empathy and safety boundaries: improving 'when to give concrete guidance.'
Key terms
- LLM-as-a-judge
- An evaluation method in which an LLM scores another model's responses against criteria. In this paper, the LLM is given the MHQR rubric and marks issues and severity for each counselor turn.
- MQM(Multidimensional Quality Metrics)
- A method from machine translation evaluation that marks errors by type and scores them with severity weights. It served as the design basis for MHQR.
- Guidance calibration
- The ability to judge when, during counseling, to stop asking questions and guide the person to concrete next steps. Excessive exploration and insufficient guidance are the typical failure types.
- Patient simulation
- A method in which an LLM plays a patient according to a profile and holds a multi-turn conversation with the model being evaluated. It is used as an alternative in mental health, where real counseling data is hard to use.
Original
Authors: John Mendonça, Daniel Tschernutter, José Pombal, Areti Vassilopoulos, Catarina Botelho, Ignacio Sarasua, Maya D'Eon, Nuno Guerreiro, et al. · Posted 2026-10-15 · License CC BY-NC-SA 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.