InteractiveWorld: A Verifiable Human-Centric Simulation Environment for Interactive MLLM Assistants

Problem

Evaluating whether wearable first-person AI assistants can actually help people requires human participants, which is expensive and hard to scale. Existing benchmarks are static, asking questions about recorded videos, so they cannot measure interaction in which the assistant's instructions change what the person does and what the scene looks like next. Robot simulators focus on controlling robot joints and are not suited to guiding humans.

Method

The authors built a three-layer environment on Isaac Sim. The physical layer uses the SMPL+H human body model (with a first-person camera on the head) and about 8,000 objects and about 50 scenes from BEHAVIOR-1K. The human action layer uses A* pathfinding and pose-adaptive manipulation to reliably perform high-level actions such as grasping and opening. The semantic layer maps natural-language instructions to executable commands through 36 action primitives. The model under evaluation acts as the assistant, choosing to stay silent, give an instruction or end the episode, while Gemini 3.1 Pro acts as the virtual human, interpreting instructions and acting on them. InteractiveBench consists of 120 tasks in six categories—guidance, planning, accessibility, decision-making, prevention and search—and success is judged automatically from object states at the end.

Results

• Even the top model, Claude Fable 5.1, had an overall task success rate of 41.7%, and all other models scored 33.3% or lower.
• On the long-horizon categories, Guidance and Planning, the best success rates were only 20.0% and 25.0%, respectively.
• Results varied widely by category. Claude Fable 5.1 scored 85.0% on decision-making tasks, while the lowest-ranked model, DeepSeek V4.1 Flash, scored 12.5% overall.
• The same virtual human scored 100% with an oracle assistant and 0% with no help, confirming that the results reflect the assistant's ability to help.
• The authors identified perception, memory and communication as the key bottlenecks.

Limitations

The human is played not by a real person but by an MLLM (Gemini 3.1 Pro), and the body is kinematically driven, so it does not fully capture the diversity and mistakes of real human behavior. With 20 tasks per category and 120 in total, the benchmark is small. Frame input is limited to 32 frames per turn. How well simulation results transfer to real wearable use has not been verified.

Why it matters

The work gives teams developing and evaluating smart-glasses-style AI assistants a repeatable, verifiable way to measure whether an assistant really helps, without user studies. The finding that even recent models fall well short on long-horizon guidance points to priorities for improvement.

Key terms

Closed-loop evaluation

An interactive form of evaluation in which the model's output changes the environment and the next observation, and that result becomes the model's input again. It contrasts with static evaluation, which answers questions about recorded video.

Egocentric assistant

An AI that sees from the same viewpoint as the user through a wearable device and gives guidance in natural language. It does not manipulate objects itself but guides the person's actions.

State-based verifiable reward

A method that automatically judges success by checking the actual object states in the simulator when a task ends (for example, whether the refrigerator is closed).

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Qinsi Wang, Wei Wen, Chong Zhou, Saksham Suri, Zechun Liu, Lemeng Wu, Hai Helen Li, Yiran Chen, et al. · Posted 2026-10-25 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.