Learning Particle-Based World Model from Human for Robot Dexterous Manipulation

zhengdong hong, Yulin Liu, Haowen Hou, Bo Ai, Jun Wang, Tongzhou Mu, Yuzhe Qin, Jiayuan Gu, Hao Su

Keywords: learning from human, world model, dexterous manipulation

Abstract: Human data constitute a rich repository of knowledge about environment dynamics and embodied interactions. Existing methods attempt to leverage this data by retargeting human motion to robots or pretraining visual representations. However, these approaches may not generalize to novel object configurations or capture the underlying dynamics of interactions. Our key insight is that the dynamics of human-object interactions are transferable to robots and can guide reinforcement learning (RL) by reducing the exploration space. Specifically, we learn a particle-based world model from human data to generate diverse high-level trajectories through model predictive control (MPC). Then those trajectories can successfully guide reinforcement learning (RL) in physical simulations by improving sample efficiency and success rate, enabling generalizable and adaptive manipulations in real-world robots after Sim-to-Real transfer. Extensive real-world experiments demonstrate our method not only outperforms baselines in higher success rate and better generalizability for single object pick-up tasks but also succeeds in challenging cluttered environments where existing methods fail, establishing an effective framework for human-guided dexterous manipulation.

Submission Number: 11