← Key papers

Paper review · November 14, 2026

SO-Arena: A Benchmark for the Capability Spectrum and Decision Reliability of System One Models

SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models

Language modelsData & evaluation Intermediate Upvotes 73 cs.CL National University of Singapore

Using 2,919 tasks across 11 domains, the benchmark measures the capability and reliability of System One models separately and shows that the top model on performance ranks only 7th on reliability.

Problem

System One models do not generate long reasoning tokens. Instead, they output a probability distribution over a fixed set of options in a single forward pass. This makes them fast for repetitive work such as routing, choosing tool actions and verifying task completion. However, existing benchmarks focus on aggregate accuracy, so they do not show well how capability differs by domain and difficulty or how consistent the decisions are.

Method

Tasks were taken from or newly built on 40 public datasets and environments, and humans and LLMs reviewed the ground-truth labels to remove questionable items. The tasks are grouped into 5 capability clusters and 11 domains, and 9 of those domains are assigned difficulty tiers using domain-specific criteria. The 2,300 static tasks and 619 environment-based tasks (337 web, 282 tool-use simulator) are packaged in a common decision format: choice, yes-no or score. The capability score C averages the top difficulty tier of each domain, with every domain weighted equally. The reliability score R is the geometric mean of two metrics. One is whether the model picks the same answer when the options are reordered 12 times (S). The other is whether it notices that there is "no correct answer" when the correct option is removed (an AUC-based N).

Results

Limitations

Difficulty tiers are defined only within each domain, so they cannot be compared across domains. Tasks that require free-form answers were excluded. Some ground-truth labels were produced by models (marked in the metadata). Stability measurements exist for only 19 of the 21 models. Several models were run with their developers' default settings, so conditions are not fully identical.

Why it matters

It provides criteria for choosing a lightweight decision model for work that needs fast, consistent decisions, such as classification and routing. It shows with numbers that choosing on accuracy alone can lead to a model that is thrown off by option order or a missing correct answer.

Key terms

System One model
A decision model that does not generate a long reasoning process and instead outputs a probability distribution over predefined options in a single forward pass.
Option-order stability (S)
The rate at which a model picks an answer with the same content even when the option order changes. It is scaled so that random choice is 0 and always giving the same answer is 1.
Missing-answer recognition (N)
A metric that uses AUC to measure whether a model gives a higher "no correct answer" probability to items whose correct option has been removed. It is calculated as 2·AUC−1.

Original

Authors: Lichen Li, Ziping Dong, Sicheng Feng, Qi Li, Xinchao Wang · Posted 2026-11-11 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.