Paper review · October 21, 2026
EgoXperience: Learning world-grounded representations from human egocentric experience
EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience
The authors built a model that jointly reconstructs the camera, scene, hands and objects from egocentric video in a single world coordinate system, along with a 50-hour automatically annotated dataset, and substantially reduced errors compared with baseline models.
Problem
Reconstructing human activity in 3D from egocentric video captured by a head-mounted camera requires camera motion, surrounding scene structure, and hand-object interactions to fit together in one consistent space. Existing methods reconstruct only some of these, so combining the outputs of multiple models requires reconciling different scales and coordinate systems. Moreover, joint annotation data for training such a unified model is hard to produce because of hand-object occlusion, fast head movement and motion blur, and usually requires expensive equipment such as multi-camera rigs or motion capture.
Method
The core is an automatic annotation pipeline called EgoX-Engine. Object meshes are pre-scanned in a dome built with 31 cameras, and real-world scale is set with AprilTags. In the video, the target object is specified with a first-frame mask, its initial pose is estimated from stereo depth and tracked with FoundationPose, and the pipeline verifies that placing the model at the estimated pose matches the observed depth. Using the camera trajectory obtained from SLAM, object, scene and hand annotations are transferred into a single session coordinate system. This produced the 50-hour EgoX-Data-6M (101 objects, 270 task scripts, with per-second captions), of which 1 hour was set aside as EgoX-Bench for evaluation. The model, EgoX, places camera, hand and object tokens on top of DINOv3 features and builds a joint representation with intra-view and cross-view attention, then transforms hands and objects into world coordinates using the predicted camera and also supervises that geometry. A scale head predicts one metric scale value per sequence.
Results
- The authors reported reducing camera error by 68.9% (HOI4D nATE), hand error by 65.6% (EgoX-Bench W-MPJPE) and depth error by 42.9% (EgoX-Bench AbsRel) compared with the strongest baseline.
- The object pose score improved by 28.0% on HOT3D AUC-S50.
- It ranked first on depth, hand and object metrics across all four benchmarks (HOT3D, EgoX-Bench, HOI4D and H2O), and had the best camera accuracy on HOI4D.
- Among the datasets in the comparison table, EgoX-Data-6M is the only one that covers indoor tabletop manipulation, moving between rooms, and outdoor manipulation while walking, and that also has all of 3D hand, 6DoF object, mesh, depth, mask and caption annotations.
- The authors described EgoX as the only method among those compared that reconstructs camera, depth, hands and 6DoF objects end-to-end in a shared world coordinate system in a single forward pass.
Limitations
Inference requires a first-frame mask of the target object, and objects are assumed to be rigid. Building the data requires a stereo headset, a 31-camera dome and pre-scanned object models, so the approach is not entirely free of equipment burden. Because the annotations themselves are outputs of existing techniques such as SLAM and FoundationPose, their accuracy is capped by those techniques. Fifty hours is small compared with large corpora such as Ego4D. The provided text is cut off partway through the experiment tables.
Why it matters
The work shows a way to automatically annotate egocentric data for robot learning without expensive motion capture. Reconstructing the camera, scene, hands and objects in one coordinate system at once removes the need for post-hoc scale alignment, making the data easier to feed directly into imitation learning or world model training. The release of the data and code also helps follow-up research.
Key terms
- Egocentric (first-person) video
- Video recorded with a camera worn on the head or body. It captures the wearer's movements and hand manipulation up close without installing external cameras.
- SLAM
- A technique in which a moving camera builds a map of its surroundings while simultaneously estimating its own position. Here it serves as the reference for bringing multiple annotations into one coordinate system.
- 6DoF object pose
- A six-degree-of-freedom value combining an object's 3D position and rotation about three axes. It describes how a manipulated object is positioned and rotated.
- MANO
- A standard 3D hand model that represents hand shape and joint pose with a small number of parameters.
Original
Authors: Xinhao Ji, Hui En Pang, Hao Li, Kairui Hu, Jiner Wang, Mingze Sun, Ziang Cao, Yalun Dai, et al. · Posted 2026-10-19 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.