Paper review · November 12, 2026
FinCosmos: an open-world evaluation framework for financial agents
FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance
The authors built an evaluation that leaves everything open: simulated users with hidden investment strategies, data the agent gathers from the web itself, and point-in-time constraints. It shows that current coding agents make "silent failures" in financial work.
Problem
Agents assigned financial tasks can fail silently. A report built on a strategy the user did not want, on unverified data, or on a backtest that used future information can look just as plausible as a correct one. Existing benchmarks fix at least one of the task, the data or the passage of time. As a result, they cannot measure these failures and instead remove them entirely.
Method
The simulated users (three investor personas) hold hidden strategies sampled from a grammar of six strategy components (stock selection, market timing, portfolio construction, risk management, execution and transaction costs), and reveal details only when the agent asks. There are about 3.8 million possible strategies. The agent is not given any data. It gathers data from the web itself, builds its own backtest while respecting the data cutoff date, and then delivers the results. Grading needs no reference answer. Programs verify point-in-time consistency and whether the reported metrics can be recomputed, and an LLM judge evaluates only semantic fidelity. Sessions are scored on 45 dimensions, and a session passes only if the user is satisfied and all 15 veto checks pass.
Results
- Across 150 sessions with four coding agents and one model without tools, not a single session passed. Quality scores before veto checks ranged from 20.2 to 37.2 out of 100.
- Of 120 agent sessions, 25 conveyed the investor's strategy correctly, 15 of those also respected the data cutoff date, and 14 had backtests free of look-ahead. Zero sessions had reported returns that could be reproduced from the trade records.
- Agents asked about only 54–64% of the details they had not been told, and got 66–83% of the rest wrong.
- Sessions the user approved (23%) and sessions that would pass by the LLM judge's standard (11%) all failed the programmatic checks. Human experts attributed 88.33% of failures to the agent, and their agreement with the LLM judge was κ=0.87.
- An agent with persistent memory that received FinCosmos feedback learned all 10 tasks it had failed on its first attempt.
Limitations
The public benchmark is small: 10 fixed tasks and 3 sessions per task and system. The simulated users are LLMs, so results may vary with the user-simulator model. The authors also cite related work on the fidelity limits of LLM simulators and supplement them with expert validation. The strategy space is limited to the defined six-component grammar. Some dimensions rely on an LLM judge.
Why it matters
Evaluating agents only by user satisfaction or an LLM judge can count plausible but wrong results as successes. This study puts numbers on that risk. It supports the case that programmatic verification and point-in-time consistency checks are needed when deploying agents in fields like finance where verification matters.
Key terms
- Look-ahead bias
- An error in which a backtest uses future information that was not available at that point in time, inflating performance.
- Point-in-time consistency
- Whether each decision used only data that was actually public on that date.
- Silent failure
- A failure that runs without errors and produces plausible-looking output, but in which something about the intent, data or timing is actually wrong.
- Veto check
- A mandatory check item: failing any one of them means failing overall, regardless of other scores.
Original
Authors: Yupeng Cao, Xueqing Peng, Lingfei Qian, Haohang Li, Yangyang Yu, Anke Xu, Wenbo Cao, Yan Wang, et al. · Posted 2026-11-05 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.