Paper review · October 23, 2026
SpatialScout: Can VLM Agents Actively Explore and Map 3D Scenes?
The authors built a benchmark in which VLM agents move a camera themselves to build a map of objects' 3D positions, and showed that even the latest models fall far short of precise mapping.
Problem
Spatial intelligence evaluation mostly relies on image and video question answering. Such evaluations leave room for guessing or answering from language priors. They also cannot check whether a model can gather observations on its own and build an accurate metric-scale 3D map.
Method
The authors placed 3,140 target objects in 80 indoor scenes (based on iTHOR, ProcTHOR and others, rendered with SAPIEN/ManiSkill3). The agent receives an object list, RGB images, and camera intrinsics and pose. It explores the scene with 10 tools, including camera movement, rotation and capture, triangulation and position submission, and estimates the center coordinates of each object. The budget is 100 model calls and 100 images. Scoring moves the ground-truth box to the predicted center and computes 3D IoU; objects that are not submitted score 0, so accuracy and coverage are measured together.
Results
- Claude Opus 5.5 scored highest, with an average IoU of 41.71% across all objects. Next were GPT-6 Astra (xhigh) at 33.16% and Claude Sonnet 5.5 at 32.26%.
- Even for Claude Opus 5.5, only 47.23% of objects exceeded an IoU of 0.5 and only 3.25% exceeded 0.9. The gap between rough placement and precise localization was large.
- Kimi-K3 (4.22%), Qwen3.8 Max (6.80%) and DeepSeek V4.1 Flash (3.48%) stayed in single digits for average IoU.
- An analysis of 40 trajectories found that the most common failure causes were incomplete coverage (26 cases) and suspected incorrect correspondences despite low triangulation residuals (21 cases).
Limitations
The benchmark covers only simulator-rendered indoor scenes and assumes that camera poses and calibration are given exactly. Each model-scene pair was run for only one episode, so run-to-run variance is not reported. Seven models were evaluated, and only object center coordinates are estimated; size and orientation are not evaluated.
Why it matters
Using VLMs in robots or embodied agents requires them to build accurate spatial maps on their own. Unlike question-answering scores, this benchmark measures actual map quality in numbers and shows the limits of current models concretely.
Key terms
- Active perception
- Instead of only looking at given images, the model decides for itself where to look next to gather the visual evidence it needs.
- Triangulation
- A technique that finds the same point in two or more images taken from different positions and computes its 3D position as the point where the lines of sight meet.
- 3D IoU
- The volume where the predicted 3D box and the ground-truth box overlap, divided by the volume of the two combined. The closer to 1, the more accurate.
Original
Authors: Zhaoxi Zhang, Hao Liang, Linxuan Wu, Yiming Wu, Yifu Yuan, Kun Liang, Chenming Tang, Meiyi Qiang, et al. · Posted 2026-10-19 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.