Paper review · October 31, 2026
Benchmarks are empirical measures over difficulty
Your Benchmark Is an Empirical Measure Over Difficulty
Treating benchmarks as distributions over a difficulty axis, the authors propose two metrics that diagnose measure mismatch and coverage gaps, and show that both problems are common across 25 benchmarks.
Problem
Benchmark scores depend not only on model performance but also on how items are distributed across difficulty levels. Many easy items mask differences between models, and if a difficulty range has no items, progress in that range does not show up in the score.
Method
The authors formalize a benchmark as an empirical measure on difficulty [0,1]. Measure mismatch is measured by the 1-Wasserstein distance to a target difficulty distribution (DDM), and coverage gaps by the maximum spacing between adjacent difficulty points. They prove theorems that DDM bounds score distortion and rank reversals, that the uniform distribution is the unique maximin target when there is no prior information, and that the maximum gap is the exact limit for detecting local improvements. Item difficulty was estimated by fitting a Bayesian 1PL IRT model to responses from 12 LLMs ranging from 1.7B to 397B (HAL results for agent benchmarks), and they also propose BenchFill, an item-rewriting algorithm that fills gaps.
Results
- They analyzed 25 benchmarks across seven domains: math, coding, agents, logic, science, instruction following and medicine.
- Ten of the 25 had a DDM of 0.10 or more relative to the uniform target, with weight concentrated in particular difficulty ranges.
- By adding only three items per benchmark, BenchFill reduced the sum of the top three difficulty gaps by 21.09–38.88%.
- The authors reported that all benchmarks passed seven pre- and post-hoc IRT diagnostics, and that leave-one-out analysis showed difficulty estimates were stable even with 12 models.
Limitations
Because difficulty is treated as a single one-dimensional axis, it cannot capture multiple dimensions of ability (comparisons with 2PL and multidimensional IRT are covered in the appendix). Difficulty estimates depend on the chosen set of models, and agent benchmarks rely on external leaderboard results. The uniform target is only a default when there is no prior information and is not always optimal. The theory requires assumptions such as Lipschitz continuity.
Why it matters
It provides a simple tool for checking which difficulty ranges a benchmark used for interpreting leaderboards or monitoring training actually measures. The released difficulty estimates can also be used directly.
Key terms
- IRT (Item Response Theory)
- A psychometric model that jointly estimates a model's ability and items' difficulty. 1PL sets the probability of a correct answer as a logistic function of "ability − difficulty".
- Wasserstein distance
- A distance that measures the difference between two distributions by how much probability mass must be moved, and how far, to turn one distribution into the other.
- Coverage gap
- A range on the difficulty axis with no items. Performance gains in this range do not show up in the benchmark score.
Original
Authors: Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, Huan Zhang · Posted 2026-10-21 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.