HexaShell: A thin harness for exploring the limits of VLM agents as robot coding agents

Problem

VLM-based agents have started to compete with learned policies on robot manipulation leaderboards. But it is not clear how far coding agents can go, or where they get stuck, when they control a robot instead of a computer. One hard part is working out positions to the centimeter from RGB pixels alone, without depth information.

Method

The harness only turns commands into joint motions and makes no decisions. The agent gets a single command-line tool called robo, video and calibration data from three RGB cameras, and a one-page manual. When an attempt fails, an optimizer agent running the same model (Codex + GPT-6-Astra, medium reasoning) reads the logs and camera frames, diagnoses the cause, and writes or fixes small pieces of tool code, including unit tests. Then it tries the same layout again. Tools can only use information the robot has observed. They compute 3D positions by triangulation: either the intersection of rays from two cameras, or the intersection of a ray with a plane already measured (the table or the top of a block). When needed, they move the wrist camera to get a second viewpoint. For each task, the tools were built on 10 layouts from one seed, then frozen and retested on 10 new layouts.

Results

• The authors report a leaderboard-style average success rate of 39.7% and a progress score of 48.0 across 42 RoboDojo tasks. This beats the best learned policies, VPP2 (32.3%, 39.3) and PhysicalRSI (31.4%, 36.3).
• The simple average success rate across 54 variants was 35.4%, higher than VPP2 at 30.6% and the same model without tools at 24.3%.
• With the same harness, model and layouts, triangulation scored 8.7 points higher than depth estimation based on Depth Anything V2. Compared layout by layout, it won 57 and lost 23.
• Results varied widely by task type. Pushing reached 92.5% and memory tasks 65.0%, but tight insertion reached only 8.3%. The authors conclude that the limit lies in contact control, not reasoning.

Limitations

The retest used 10 new layouts from the same seed per task, with one episode each. Leaderboard entries are evaluated on 3 seeds with 50 episodes each, so the evaluation setups are not the same. The retest prompts also included task descriptions, the scoring table and notes from the development phase, and in 25 tasks the optimizer read notes on checker requirements. An episode takes a median of 739 seconds, but most of that is model latency, which is not counted in the benchmark cost. Evaluation was done only in simulation, and performance is low on contact-rich precision manipulation.

Why it matters

The work shows that a general-purpose coding agent can compete with learned policies just by turning failures into code over time, without collecting robot data or training a model. It also gives concrete numbers showing that contact-rich precision manipulation is the current bottleneck for VLM agents.

Key terms

Harness

An execution framework that wraps a model and connects it to an environment. The harness in this paper is kept thin: it only turns commands into robot motions and makes no decisions.

Triangulation

A method for finding a 3D position as the point where the lines of sight from two calibrated cameras meet when both see the same point. It can measure positions from RGB images alone, without a depth sensor.

Active Perception

An approach in which the agent moves the sensor itself to get better observations. In this paper, the wrist camera is moved to get a second viewpoint, which uses up action steps.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Hongcheng Gao, Alex Zhou, Jay Zhu, Zelin Zheng, Shijia Ge, Xiping Li, Di Wu, Yazhe Wang, et al. · Posted 2026-10-16 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.