Paper review · November 1, 2026
What the Timeline Says: Evidence-Grounded Significance Testing of Temporal Clinical Hypotheses
Testimony from Timelines: Evidence-Grounded Significance Testing of Temporal Clinical Hypotheses
The authors built STRATA, a benchmark of 8,735 temporal hypotheses over vaccine adverse event timelines with step-by-step ground truth, and ART, a staged verification method, revealing that LLMs make the most mistakes in choosing which records to count as evidence.
Problem
Population-level temporal hypotheses such as 'after vaccination, symptom A appears before B' can only be decided by significance testing on the correct records. Verification requires setting the null hypothesis, choosing the test, and deciding which records to count in the denominator and numerator. Even if one of these is wrong, the conclusion can still be right by chance. Yet existing LLM data analysis benchmarks hand over the relevant data in advance and grade only the final conclusion.
Method
STRATA generates 8,735 hypotheses over clinical timelines built from 3,166 VAERS reports in MedTempo (three COVID-19 vaccines), using 26 template groups, 7 hypothesis types (precedence, co-occurrence, first/last, position, proximity, threshold, etc.), 3 difficulty levels and 4 writing styles. Each hypothesis comes with ground truth for the plan (null hypothesis, test, denominator rule), the evidence set (denominator D, numerator N), the p-value and the verdict (supported/refuted/insufficient subtypes). The ground truth is computed mechanically from published codebook rules and reproduced by a deterministic oracle. Facts whose verdict is significant in one direction are also posed as hypotheses in the opposite direction, to catch systems that support both. ART (Articulate, Retrieve, Test) 1) expresses the hypothesis as a plan, 2) retrieves the evidence set matching the plan, and 3) runs the specified test deterministically. Steps 1 and 2, handled by the model, include up to two verify-and-revise loops, and a contract approach is used in which the harness resolves each element of the plan into exact rules.
Results
- On a test set of 1,500 items, ART with gemma4-31B reached a verdict accuracy of 0.981, higher than the EHRSQL single-query baseline (0.634) and EHRAgent (0.733). It was within 0.01 of Opus 5.5 (0.991) and GPT-5.6 (0.981) using the same method.
- With Qwen2.5-32B as well, ART's accuracy was 0.806, higher than EHRAgent (0.596) and EHRSQL (0.528).
- The authors reported that the main failure point of the systems is the step of choosing which records to count as evidence. This is the 'verdict–evidence gap', which verdict accuracy alone does not reveal.
- Frontier models performed well when given the codebook. On EHRSQL-C, which provides the codebook, Opus 5.5 scored 1.000 and GPT-5.6 scored 0.984.
- Ablation (gemma4-31B): accuracy dropped to 0.779 when the model made the verdict itself, 0.837 without the verify-and-revise loop, and 0.947 when a narrative plan was used instead of the contract.
Limitations
The data is limited to 3,166 VAERS reports for three COVID-19 vaccines. Even supported hypotheses reflect only regularities in what was reported and do not imply causal effects. Hard hypotheses comparing two cohorts are few, at 87 (82 in the test set). Hypotheses are generated mechanically by filling template slots and may not capture the full variety of real claims on the web. The ground truth depends on codebook conventions (e.g. how same-stage events are handled). Due to API costs, some frontier model experiments and ablations were not run.
Why it matters
It provides an evaluation framework for checking whether the 'plausible statistical conclusions' LLMs produce are right for the right reasons, in fields like medicine where wrong decisions are costly. It exposes evidence selection errors that existing evaluations focused only on final accuracy miss. It shows that a design that splits decisions into steps and runs tests deterministically is enough for a 31B open model to come close to frontier models.
Key terms
- Temporal Hypothesis Verification
- A task that takes a claim about the order or timing of events across multiple patient timelines and, using a null hypothesis and a significance test, decides whether it is supported, refuted or insufficient.
- Denominator/Numerator Sets
- The set of reports eligible for judging a hypothesis (D) and, within it, the set of reports that support the hypothesis (N). For example, for a 'precedence' hypothesis, only reports where the two events are at different stages are included in the denominator.
- Verdict–Evidence Gap
- A case where the final verdict is correct but the set of records used as its evidence is wrong. Accuracy metrics alone do not reveal it.
- Codebook
- A rulebook that sets out, in text, the conventions for turning a natural-language hypothesis into a testable population and test. It can be given to the system or withheld.
Original
Authors: Chengyang He, Wenlong Zhang, Koduvayur Subbalakshmi, Ping Wang · Posted 2026-10-29 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.