Paper review · October 16, 2026
Player-1: An Agent That Learns to Play Games from Synthetic Experience
Player-1: Learning to Play from Synthetic Experience
It extends the action space of a computer-use agent for games and applies reinforcement learning in 16 synthetic environments, substantially improving control performance even in games not used for training.
Problem
Computer-use agents can read the screen and operate a keyboard and mouse. However, they are still weak at games that require precise aiming, continuous movement and simultaneous inputs. Demonstration data covers only part of the states a game can produce, and it is also hard to attach programmatic rewards to commercial games to build large-scale interactive training environments.
Method
First, they added relative pointer movement, key holding (hold) and composite actions that execute multiple inputs simultaneously to the action space. Next, they converted 6,044 gameplay trajectories (295,097 turns) to this action space, used a VLM to attach task instructions and step-by-step visual reasoning, and performed fine-tuning (SFT). They then built 16 2D and 3D synthetic environments (Player-1-Gym) across five families: aiming, combat, driving, navigation and UI control. These environments give verifiable rewards based on game state and randomize appearance, camera, scene and control sensitivity. Finally, they jointly trained Qwen3.5-9B and 35B-A3B with PPO using per-environment advantage normalization.
Results
- On MineDojo-300 success rate, Player-1-9B rose from 10.54% at the SFT stage to 38.33%. This is higher than the strongest external baseline, GameWAM (12.63%).
- Player-1-35B-A3B rose from 23.38% after SFT to 41.04% on MineDojo-300 (+17.7 pp).
- For the 9B model, the number of tasks succeeded on at least 4 of 8 attempts rose from 20 to 136 out of 300.
- The authors report that performance also improved over SFT on ViZDoom-10 and GameWorld, which were not among the reinforcement learning training environments, and that transfer improved as more diverse synthetic environments were used.
Limitations
Reinforcement learning ran for 200 steps, and the synthetic environments are only the 16 the authors built. External baselines were measured under the authors' evaluation protocol. Success rates on MineDojo Harvest tasks remain low, around 10–12%, even after reinforcement learning. The provided text was cut off, so the final ViZDoom and GameWorld numbers and detailed ablation results could not be checked.
Why it matters
It provides evidence that training that transfers to other games is possible using only synthetic environments targeting common control skills, without building rewards for each real game. It is a useful reference for teams looking to extend computer-use agents to domains that require real-time control.
Key terms
- Computer-Use Agent (CUA)
- An agent that performs digital tasks by looking at screenshots and outputting keyboard and mouse actions.
- Composite Action
- An action that executes multiple keyboard and mouse inputs simultaneously within one control step. For example, pressing Space while holding W to jump while running.
- RLVR
- Reinforcement learning that uses rewards that can be verified programmatically. Here, changes in game score are used as step-by-step rewards.
- Environment Randomization
- A technique that randomly changes environment factors such as color, camera, layout and control sensitivity during training so the model does not overfit to a specific environment.
Original
Authors: Mingyu Ouyang, Yiran Wang, Siyuan Hu, Xingyilang Yin, Huiqiang Sun, Guangzhi Wang, Junfu Pu, Cheng Cheng, et al. · Posted 2026-10-14 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.