Paper review · October 13, 2026
Reflex: Vision-Grounded Reactive Whole-Body Control for Humanoids
Reflex: Visually Grounded Reactive Humanoid Control
By splitting training into three stages (whole-body control, object-dynamics reasoning and first-person visual grounding), the authors got a real Unitree G1 to catch boxes thrown by people using only onboard RGB-D.
Problem
Reacting with the whole body to an incoming object within one second requires whole-body coordination, real-time dynamics prediction under latency, and perception from a moving first-person camera, all at once. Existing systems relied on end-effectors, motion capture or marked objects and solved only some of these.
Method
Using box catching as the representative task, training proceeds in three stages. In stage 1, a whole-body catching Teacher policy is trained with the simulator's box state as input, using PPO and a human motion prior (AMP). In stage 2, a privileged encoder is built that compresses box observations with 20–60 ms latency and dropouts into an 8-dimensional dynamics latent vector. It is initialized by matching Teacher actions, then data is collected DAgger-style and refined with PPO. In stage 3, with the controller frozen, a visual encoder is trained to regress this latent vector from RGB-D history and camera ego-motion alone.
Results
- In simulation, the catch success rate from first-person RGB-D was 85.7%, close to the privileged-observation baseline that receives the box state directly (87.9%).
- Removing temporal visual history lowered success by 35.6 percentage points for frontal throws and 39.8 percentage points for throws with a 0.6 m lateral offset.
- The real Unitree G1 caught 65% and 40% of boxes thrown by people from 2.5 m and 5 m, respectively. That is 76% and 59% of human success rates at the same distances (85.0% and 68.3%).
- Action-matching initialization outperformed weight-mapping initialization by 5.2 percentage points on average. Regressing the learned latent vector worked better than regressing the full box state.
Limitations
There is only one task, box catching. Real-robot success drops sharply as distance increases. The hardware's median camera-to-control latency (69 ms) exceeds the latency range used in training (20–60 ms). Some comparisons (such as the effect of latency training) are single-seed results.
Why it matters
It demonstrates on real hardware a humanoid that reacts with its whole body to fast-moving objects using only onboard vision, without external equipment such as motion capture. The design that separates perception from control through a latent vector could extend to other reactive skills.
Key terms
- Privileged Learning
- A method in which a Teacher policy is trained on exact states available only in a simulator, and a Student that uses only real sensors learns by imitating it.
- AMP(Adversarial Motion Prior)
- A technique in which a discriminator compares movements with human motion data and rewards 'human-like' motion, helping the robot learn natural movements.
- DAgger
- An imitation learning method that grows the dataset by collecting the Teacher's correct actions at states the policy being trained actually visits.
- Ego-motion
- The movement of the robot that carries the camera. Because it mixes with object motion in the video, it must be provided as a separate input for object dynamics to be estimated correctly.
Original
Authors: Taoyang Jia, Weikai Huang, Linxin Song, Jared Darlington, Jieyu Zhao, Yue Wang, Jiafei Duan, Zhongzheng Ren, et al. · Posted 2026-10-08 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.