Paper review · November 14, 2026
SO-Arena: A Benchmark for the Capability Spectrum and Decision Reliability of System One Models
SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models
Using 2,919 tasks across 11 domains, the benchmark measures the capability and reliability of System One models separately and shows that the top model on performance ranks only 7th on reliability.
Problem
System One models do not generate long reasoning tokens. Instead, they output a probability distribution over a fixed set of options in a single forward pass. This makes them fast for repetitive work such as routing, choosing tool actions and verifying task completion. However, existing benchmarks focus on aggregate accuracy, so they do not show well how capability differs by domain and difficulty or how consistent the decisions are.
Method
Tasks were taken from or newly built on 40 public datasets and environments, and humans and LLMs reviewed the ground-truth labels to remove questionable items. The tasks are grouped into 5 capability clusters and 11 domains, and 9 of those domains are assigned difficulty tiers using domain-specific criteria. The 2,300 static tasks and 619 environment-based tasks (337 web, 282 tool-use simulator) are packaged in a common decision format: choice, yes-no or score. The capability score C averages the top difficulty tier of each domain, with every domain weighted equally. The reliability score R is the geometric mean of two metrics. One is whether the model picks the same answer when the options are reordered 12 times (S). The other is whether it notices that there is "no correct answer" when the correct option is removed (an AUC-based N).
Results
- 21 models were evaluated. The System Two reference models GPT-6 Sol (82.3) and GPT-6 Luna (79.7) ranked 1st and 2nd on the capability score. The top System One models, GLiDE and KnowLine-31B (70.2), scored about 10 points below Luna.
- System One models were far ahead on speed. All open-weight System One models answered in under 0.3 seconds per task on a single H100. The authors reported that they are tens of times faster than GPT-6 Sol and Luna.
- The System One model ranked 1st on capability placed only 7th on reliability. JevAny-27B ranked 11th on capability but had the highest reliability among the System One models evaluated.
- Performance fell as difficulty increased, and the drop was especially large on agent tasks. The authors reported that the gains from scaling model size or adding reasoning vary widely by domain.
Limitations
Difficulty tiers are defined only within each domain, so they cannot be compared across domains. Tasks that require free-form answers were excluded. Some ground-truth labels were produced by models (marked in the metadata). Stability measurements exist for only 19 of the 21 models. Several models were run with their developers' default settings, so conditions are not fully identical.
Why it matters
It provides criteria for choosing a lightweight decision model for work that needs fast, consistent decisions, such as classification and routing. It shows with numbers that choosing on accuracy alone can lead to a model that is thrown off by option order or a missing correct answer.
Key terms
- System One model
- A decision model that does not generate a long reasoning process and instead outputs a probability distribution over predefined options in a single forward pass.
- Option-order stability (S)
- The rate at which a model picks an answer with the same content even when the option order changes. It is scaled so that random choice is 0 and always giving the same answer is 1.
- Missing-answer recognition (N)
- A metric that uses AUC to measure whether a model gives a higher "no correct answer" probability to items whose correct option has been removed. It is calculated as 2·AUC−1.
Original
Authors: Lichen Li, Ziping Dong, Sicheng Feng, Qi Li, Xinchao Wang · Posted 2026-11-11 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.