Egocentric data for robot learning: what recent work shows
Short answer: egocentric data is video and motion recorded from a person's own point of view while they do a task. It can be collected in many more places than robot teleoperation, and recent work uses it at large scale, mostly in combination with robot data. The practical setup is a human-data pipeline for volume plus a few real robot stations for training on robot data and testing policies.
What the recent work shows
- EgoVerse (2026) is a shared platform and dataset with 1,362 hours of egocentric human demonstrations, from Georgia Tech, Stanford, UC San Diego, ETH Zürich and industry partners. Both of its robot setups use two 6-DoF ARX5 arms (paper).
- EgoWAM (Georgia Tech, 2026) trains manipulation policies on egocentric video by predicting both the actions and how the scene will change (project).
- UMI (2024) records demonstrations with handheld grippers instead of a robot. Its deployment code for ARX arms is listed in Stanford's arx5-sdk (UMI).
- Xiaomi-Robotics-1 (2026) was pre-trained on more than 100,000 hours of embodiment-free UMI data, then post-trained on more than 7,200 hours of in-house robot data (project).
- SIM1 (2026) turns about 200 human demonstrations into synthetic cloth-manipulation data in a physics-aligned simulator (project).
The pattern across these projects: human data supplies volume and variety, and robot data on real arms grounds the policy in the robot's own body.
Where it helps and where it does not
| Helps with | Still needs robot data for |
|---|---|
| Many scenes, objects and people | The robot's own kinematics and gripper |
| Long-tail tasks that are rare in the lab | Contact and force behaviour |
| Collecting where a robot can't go | Final evaluation of the policy |
A practical setup
- Record egocentric demonstrations for breadth. ARX's AC 2, released in February 2026, is ARX's system for egocentric (EGO) data and reinforcement learning (RL).
- Keep bimanual robot stations for robot data and testing. The bimanual platform and AC one record leader–follower demonstrations on ARX arms.
- Test on the same arms other labs use. EgoVerse's robot setups use ARX5 arms, which makes comparison easier.
