Paper review · October 27, 2026
HiSTEVE 1.0: A Human-Interface VLA Agent for Open-World Games
HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games
By combining a harness with a VLM planner, structured memory and a completion detector with a VLA that operates directly through keyboard and mouse, the system achieved top performance on atomic tasks in Minecraft and carried out long-horizon tasks.
Problem
An agent that plays games like a human needs both long-horizon autonomy and control through native interfaces (keyboard and mouse). Existing Minecraft agents either do only visuomotor control with weak long-horizon planning, or plan long-horizon but rely on environment APIs or predefined skills for execution. Deciding when a subtask has ended and interference between GUI and non-GUI actions are also difficult.
Method
The architecture is hierarchical. The upper-level harness has the VLM planner choose the next task only within a schema of atomic subtasks the VLA can execute. Four types of memory (task, episodic, procedural and world) track verified progress. When a completion-state detector that takes the VLA policy context as input flags a completion candidate, the harness verifies it and then decides whether to proceed, recover or replan. The lower-level VLA (based on Qwen2.5-VL-7B) sees only the first-person screen and generates keyboard and mouse action tokens at 20Hz. RS-Norm, which uses separate RMSNorm scales for GUI and non-GUI actions, and Dual-Mouse, which separates mouse-movement tokens, reduce interference. Training has three stages: world-knowledge injection, imitation learning, and reinforcement learning from environment feedback (GRPO and a DPO-style pairwise loss).
Results
- The authors report top performance on atomic tasks, with 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA.
- Against the strongest baseline in the main comparisons, the average success rate was 20.0 percentage points higher on MineStudio Bench and 0.6 points higher on OpenHA, and on LongBench it improved by up to 28.6 points depending on the task group.
- With the VLA held fixed, the full harness achieved more than twice the goal success rate of a fixed-length handoff approach.
- The trained completion detector judged completion more accurately than a prompted general-purpose VLM, and the agent also completed the long-horizon task of crafting an iron pickaxe from scratch.
Limitations
Experiments are limited to a single environment, Minecraft. The improvement over the baseline on OpenHA is small at 0.6 percentage points. The authors state that better atomic-task execution does not raise long-horizon task performance by a comparable amount. The atomic subtask schema has to be designed by humans.
Why it matters
This is a concrete example of designing an agent that performs complex tasks without APIs, using only the same screen and inputs a human uses. The "harness" design linking high-level planning to low-level execution, and the techniques for reducing interference between heterogeneous control modes, are also relevant to computer-use agents and robots.
Key terms
- VLA(Vision-Language-Action)
- A model that takes images and language instructions and directly outputs actions (here, keyboard and mouse inputs).
- Agent harness
- An orchestration layer outside the model calls that coordinates planning, memory, completion verification and recovery.
- RS-Norm(Regime-Specific RMSNorm)
- A method that uses different RMSNorm scale vectors for GUI actions and for ordinary movement and camera control. It reduces interference between the two modes with only a small increase in parameters.
- Negative transfer
- A phenomenon in which, when tasks of different kinds are trained together, learning one task actually lowers performance on another.
Original
Authors: Zecong Tang, Zian Guan, Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Bo Dai · Posted 2026-10-22 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.