Paper review · October 20, 2026
LiveMACE: A Process-Aware Benchmark for Evaluating LLM Agent Capabilities in Changing Markets
LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets
The authors diagnosed LLM agents' tool use, memory, rule compliance and collaboration from execution logs in a live financial-market paper-trading environment, and showed that returns and actual capabilities often diverge.
Problem
Evaluating agents on outcomes alone does not reveal which capabilities produced those outcomes. In environments where external conditions keep changing, the problem grows, because outcomes are a mix of agent behavior and environmental change.
Method
Five frontier LLMs were run for about 30 days (2026-04-13 to 05-13) with irreversible paper-trading accounts in live markets for 6 cryptocurrencies and 17 US stocks. A decision round is held every 4 hours. On a shared ReAct framework, the authors attached matched Base, tool use, persistent memory, rule compliance and multi-agent collaboration configurations. Alongside cumulative returns, they derived mechanism-specific metrics from the full decision logs, including Tool Call Score (TCS), Memory Usage Quality (MUQ), Rule Satisfaction Score (RSS) and Observable Collaboration Score (OCS), computed from a mix of LLM-as-a-Judge and programmatic objective metrics.
Results
- The authors report that cumulative-return rankings shifted substantially over time. Pairwise rank reversals numbered 21, 19, 19 and 29 in the tool, memory, rule and collaboration configurations respectively, while reversals in capability-metric rankings numbered 0, 5, 8 and 2.
- A different model led each capability: GPT-5.4 in tool use, Grok-4.20 in memory use, Gemini-3.1-Pro in rule compliance and DeepSeek-V3.2 in collaboration. Grok-4.20 had the lowest TCS yet the highest return in the tool configuration.
- All models gathered sufficient evidence, with information coverage (IC) of 0.809–0.911, but invocation efficiency (IE), at 0.581–0.731, was the weakest item.
- In the memory configuration, GPT-5.4 ran 332 retrievals but stored only one memory. Gemini accumulated 93 memories but had the lowest MUQ, at 0.497. All five models with memory had lower tail loss than in the configuration without memory.
Limitations
The evaluation is limited to about 30 days, 5 models and 23 assets, and uses paper trading rather than real trading. Some metrics rely on LLM judge verdicts. Because returns depend heavily on market conditions during that period, the results are hard to generalize to other periods.
Why it matters
When deploying agents, looking only at final performance metrics does not show which mechanisms actually work. An evaluation design that separately measures mechanism access, effective use and performance can be applied to monitoring in production.
Key terms
- Outcome–Capability Gap
- A phenomenon in which the final outcome (returns) and measurements of the capabilities that produced it (tool use, memory, etc.) disagree.
- ReAct
- An agent framework in which an LLM solves tasks by alternating between reasoning (Reason) and acting (Act, such as tool calls).
- Process-aware evaluation
- A method that evaluates agents based not only on the final outcome but on the full execution log of observations, reasoning, tool calls and actions.
Original
Authors: Jun Zhao, Leiming Fu, Yanbo Wen, Yiding Wang, Xuantong Liu, Yang Shu, Yuyang Lu, Xuanran Xing, et al. · Posted 2026-10-07 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.