RealCLI: a unified benchmark for evaluating human–agent collaboration in CLI agents

Problem

CLI agents are carrying out increasingly long tasks, but existing benchmarks either assume user instructions stay fixed throughout execution or evaluate only the final result. In practice, users add information mid-run, change constraints and decide whether to approve risky operations. Yet no unified evaluation measured whether agents act in ways that are easy to understand, follow users' course corrections and honestly report risks during this process.

Method

Grounded in HCI theory, collaboration quality is divided into five dimensions: legibility, steerability, observability and recovery, trust calibration, and safety and risk awareness. Based on analysis of real CLI logs, these are refined into 22 sub-dimensions and 51 interaction scenario templates. Each of 101 tasks spanning 10 categories and 52 subdomains is evaluated in two modes. EXEC provides the full specification up front, while in COLLAB a scripted user gives information in pieces and issues probes such as corrections, constraint changes and approvals. Task success is judged by the same verifier, and collaboration quality is aggregated over 1,658 checklist items in four levels. The final metric, J_collab, is the product of the collaboration-mode task success rate and the collaboration score.

Results

• The authors evaluated 11 models × 3 harnesses (Claude Code, Terminus-2, Codex), 33 configurations in total. The best configuration (Claude Opus 5) had an average score of 49.2%.
• The highest J_collab was 37.6% for Claude Opus 5 + Terminus-2, 27.9 percentage points below the 65.4% EXEC success rate of the same configuration.
• The authors report that user interaction made virtually no difference to the overall pass rate, and that its effect varied by task type and model.
• Agents did relatively well on legibility and on observability and recovery, but were consistently weak on steerability, trust calibration, and safety and risk awareness. Even models with the same task success rate differed widely in collaboration traits.

Limitations

The user is simulated by script and cannot reflect the full range of real user reactions. With 101 tasks, the scale is limited. Collaboration scores may be sensitive to the design of the checklist and the threshold (75% scenario pass rate). The text reports the best average score inconsistently: 49.2% in the abstract and tables, and 52.7% in the introduction.

Why it matters

When deploying coding and operations agents in real work, collaboration quality matters alongside task success rate: accepting user intervention, disclosing risks and seeking approval before destructive operations. This benchmark provides a standard for comparing those differences quantitatively.

Key terms

CLI agent

An LLM agent that carries out tasks by directly handling code, files and system resources through terminal commands. Examples include Claude Code and Codex.

Paired evaluation (EXEC/COLLAB)

A design that runs autonomous execution and user-collaboration execution separately on the same task, environment and verifier, and compares them directly.

Trust calibration

An agent's ability to report the limits of its own capabilities, risks and actual progress accurately, without exaggerating or hiding anything.

J_collab

A metric that multiplies the collaboration-mode task success rate by the collaboration quality score. If either one is low, the overall score is low as well.

Original

• arXiv abstract
• PDF
• HTML full text
• Hugging Face

Authors: Yunqi Liu, Peiyu Wang, Bingli Wang, Leo Luo, Xuanwu Song, Xudong Ma, Hongxin Li, Yaxin Du, et al. · Posted 2026-10-15 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.