Paper review · October 17, 2026
RoTA-Bench: Evaluating 3D Spatial Reasoning Through Interactive Projection Alignment
RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignment
The authors evaluated the 3D spatial reasoning of VLMs with an interactive task in which an object is rotated to match a target silhouette, and found a large gap between the best model's 17% success rate and humans' 100%.
Problem
Existing spatial reasoning benchmarks are question answering on fixed images, so they involve no actions or feedback. Interactive benchmarks lean toward high-level planning or pathfinding on grids. As a result, whether VLMs can understand 3D orientation from 2D observations and turn it into precise actions has not been measured.
Method
The authors inflated 2D icons into sphere-like shapes to automatically generate unfamiliar 3D objects whose front silhouette matches the original icon. The initial pose is created by randomly applying 30° rotations 6–16 times to the object. Applying the rotations in reverse returns it to the original pose, so every task is guaranteed to be solvable. The model looks at the target silhouette, the current silhouette and a depth-shaded image, and chooses one of 24 rotations made of 3 axes × 2 directions × 4 step sizes (1°/5°/10°/30°). A task succeeds if projection IoU reaches 0.90 or higher within at most 48 decisions. Of 674 tasks, 100 were selected by stratified sampling as the test set. Diagnostic metrics such as 3D pose error, the share of effective directions chosen and the share of optimal angles chosen were also measured.
Results
- GPT-5.6-Sol had a success rate of 17% (final IoU 0.598), while 20 undergraduate students succeeded 100% of the time over 400 attempts (final IoU 0.923).
- Claude Opus 5, Claude Sonnet 5, Qwen 3.7 Plus and SpatioLM-InternVL3.5 had a 0% success rate. A ReAct agent configuration reached 16%.
- Compared from the same states, the share of actions that improved silhouette overlap but worsened the actual 3D pose was 19.4% for models and 10.9% for humans.
- Without the interaction history, 96.8% of model actions concentrated on the largest 30° rotation, and the success rate dropped sharply.
Limitations
The objects are synthetic shapes made by inflating icons, the projection is orthographic, and actions are simplified to 24 discrete rotations, which differs from real robot manipulation environments. The test set has 100 tasks, and the human participants were 20 undergraduate students. Because success is judged by an IoU threshold of 0.90, a symmetric object may count as a success even when pose error remains.
Why it matters
This shows that for using VLMs to control robots or 3D tools, 'the ability to recognize images' and 'the ability to precisely manipulate 3D orientation' are different. The diagnosis that history management and fine adjustment are weak points can be applied directly to agent design.
Key terms
- Projection IoU
- The overlapping area of the object's current 2D silhouette and the target silhouette divided by their combined area. The closer to 1, the better the match.
- Orthographic projection
- A way of drawing a 3D object straight onto a plane without perspective. Size does not change with distance.
- Closed-loop evaluation
- An evaluation method that repeats a cycle in which the model acts, receives the updated observation and chooses its next action.
Original
Authors: Qingchuan Ma, Yuexiao Ma, Xiang Meng, Yixuan Zou, Tianyu Xie, Wang Chen, Yuhui Zeng, Yuhang Wu, et al. · Posted 2026-10-13 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.