← Key papers

Paper review · November 12, 2026

FinCosmos: an open-world evaluation framework for financial agents

FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance

AgentsData & evaluationFinance & insurance Intermediate Upvotes 20 cs.AI The Fin AI

The authors built an evaluation that leaves everything open: simulated users with hidden investment strategies, data the agent gathers from the web itself, and point-in-time constraints. It shows that current coding agents make "silent failures" in financial work.

Problem

Agents assigned financial tasks can fail silently. A report built on a strategy the user did not want, on unverified data, or on a backtest that used future information can look just as plausible as a correct one. Existing benchmarks fix at least one of the task, the data or the passage of time. As a result, they cannot measure these failures and instead remove them entirely.

Method

The simulated users (three investor personas) hold hidden strategies sampled from a grammar of six strategy components (stock selection, market timing, portfolio construction, risk management, execution and transaction costs), and reveal details only when the agent asks. There are about 3.8 million possible strategies. The agent is not given any data. It gathers data from the web itself, builds its own backtest while respecting the data cutoff date, and then delivers the results. Grading needs no reference answer. Programs verify point-in-time consistency and whether the reported metrics can be recomputed, and an LLM judge evaluates only semantic fidelity. Sessions are scored on 45 dimensions, and a session passes only if the user is satisfied and all 15 veto checks pass.

Results

Limitations

The public benchmark is small: 10 fixed tasks and 3 sessions per task and system. The simulated users are LLMs, so results may vary with the user-simulator model. The authors also cite related work on the fidelity limits of LLM simulators and supplement them with expert validation. The strategy space is limited to the defined six-component grammar. Some dimensions rely on an LLM judge.

Why it matters

Evaluating agents only by user satisfaction or an LLM judge can count plausible but wrong results as successes. This study puts numbers on that risk. It supports the case that programmatic verification and point-in-time consistency checks are needed when deploying agents in fields like finance where verification matters.

Key terms

Look-ahead bias
An error in which a backtest uses future information that was not available at that point in time, inflating performance.
Point-in-time consistency
Whether each decision used only data that was actually public on that date.
Silent failure
A failure that runs without errors and produces plausible-looking output, but in which something about the intent, data or timing is actually wrong.
Veto check
A mandatory check item: failing any one of them means failing overall, regardless of other scores.

Original

Authors: Yupeng Cao, Xueqing Peng, Lingfei Qian, Haohang Li, Yangyang Yu, Anke Xu, Wenbo Cao, Yan Wang, et al. · Posted 2026-11-05 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.