Paper review · October 10, 2026
Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?
The authors built a benchmark of 20 text games whose rules are hidden or run counter to common sense, measuring LLM agents' ability to learn through interaction rather than from prior knowledge.
Problem
Many tasks in existing self-evolving agent benchmarks either state the rules in the instructions or are already familiar from pretraining. So even when performance rises, it is hard to tell whether the agent learned something new through interaction or reasoned from knowledge it already had.
Method
The benchmark consists of 20 newly designed game templates, each providing deterministic feedback and automatic scoring. Ten use the same instance every episode, and ten keep the rules but change characters and layout across instances. The 5 hardest games are played for 10 episodes and the rest for 5, measuring maximum score (Max), average (Mean), improvement from first to last (LG) and learning slope (LS). The authors compared 11 backbones with the OpenCode harness, compared Memory, Reflexion, EvoTest, ReasoningBank, AWM and Naive with the backbone fixed, and also compared against human players.
Results
- With the OpenCode harness, Claude Opus 5.5 scored highest with Max 80.1 and Mean 61.0, while Qwen 3.8 Max showed the largest improvement with LG +25.2 and LS +6.30.
- With Kimi K3, Memory, which feeds in past interaction traces as-is, led methods that summarize experience, such as Reflexion (61.6, +14.8) and AWM (56.1, +4.8), on every metric, with Max 63.3 and LG +16.9.
- On the same backbone, the OpenCode harness outperformed the self-evolving methods evaluated (Kimi K3: OpenCode Max 68.5 vs. 63.3 for the best method, Memory).
- The top human player's Max was 84.3, higher than the best agent (80.1), and the authors report that humans tried more varied strategies and repeated actions less.
- The authors report cases where agents worked out the rules but failed because they could not plan ahead (Haunted Inn), and cases where they got stuck on partially working strategies (GemForge).
Limitations
Each game template was evaluated with one seed and 3 trials, so the statistical scale is small. It is limited to text games, and whether results generalize to real work environments has not been confirmed. The human comparison selects the top k players by Max. Rankings may change depending on the aggregation method, such as weighting.
Why it matters
It is a tool that measures the 'ability to learn by trial and error' needed when deploying agents to unfamiliar systems, separately from prior knowledge. It shows in practical terms that preserving raw traces may work better than summarizing experience, and that harness design matters as much as the backbone.
Key terms
- Self-evolving Agent
- An agent that uses experience from earlier attempts in later attempts through memory, reflection, rule extraction and similar means, without changing model weights.
- Harness
- An agent execution framework that wraps a model and provides tool calling, memory management and an execution loop. In this paper, OpenCode is the main example.
- Learning Gain / Learning Slope
- The score improvement of the last few episodes over the first few, and the slope of a line fitted to per-episode scores. They measure how much an agent learns.
Original
Authors: Yibo Li, Jinhang Qiu, Zhi Zheng, Qianyun Guo, Jiaying Wu, Shuo Ji, Bryan Hooi · Posted 2026-10-06 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.