Paper review · October 20, 2026
Learning to think and explore in a 3D world
Learning to Think and Explore in 3D World
The authors used reinforcement learning to train a VLM that sees only one image at a time to move its own viewpoint inside a 3D scene. It reached an average accuracy of 53.2% across 6 3D benchmarks.
Problem
Existing 3D vision-language models assume they receive complete observations up front, such as multi-view images, video or point clouds. But when only part of a scene is visible, a model needs not only to reason with the information it has, but also to go and find the information it is missing.
Method
The 3D scene is treated as an interactive environment rather than a model input. The model starts from a single first-person image and a question, reasons, takes movement and rotation actions (e.g. move left) to receive new rendered images, repeats this, and then answers. The process is formalized as a POMDP. The model is first given supervised fine-tuning on 5,712 trajectories generated with GPT-5.4 to learn the interaction format, then trained with GRPO reinforcement learning on 51,543 questions. The reward is mainly whether the final answer is correct (LLM judge, exact match), plus an interaction reward given only for correct answers and a format reward. Which viewpoints to look at was not directly supervised.
Results
- The authors report an average accuracy of 53.2% across 6 3D benchmarks, 22.0 percentage points higher than the base Qwen model and 1.0 percentage point higher than Gemini 3.6 Flash.
- It was more accurate than existing open-source 3D methods while using 50–70% fewer input frames.
- On 4 benchmarks from domains not used in training, it averaged 5.9 percentage points higher than the initial supervised fine-tuned model.
- The gain over the base model was 0.6 percentage points on questions answerable from the initial view alone, but grew to 16.3 percentage points on questions that require a lot of exploration. The authors interpret this as the model learning to adjust how much it explores to fit the question.
Limitations
Embodiment is simplified to directly controlling a camera inside scanned indoor scenes. There is no collision checking, so the camera can pass through objects. The supervised trajectories were generated by GPT-5.4, and part of the reward is judged by an LLM judge, so judge errors may affect training. Evaluation focused on indoor 3D QA datasets.
Why it matters
The paper shows that a VLM can learn on its own "what to look at more, and when" even when rewarded only on whether the final answer is correct. This is useful when designing visual reasoning models for robots or agents that need to gather information actively.
Key terms
- POMDP
- A framework for decision problems in which an agent cannot see the full state of the environment and must choose actions from partial observations.
- GRPO
- A reinforcement learning method that samples several rollouts from the same input and standardizes rewards within the group to compute relative advantages. It does not need a separate value model.
- Rollout
- A record of one full run produced by a policy as it interacts with the environment from start to finish. Here, it is the whole process of reasoning, moving and observing until an answer is given.
Original
Authors: Yijiang Li, Bingyang Wang, Nuno Vasconcelos · Posted 2026-10-15 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.