One-shot Learning for Robot Manipulation through Egocentric Video Demonstration

Xiwen Dengxiong, Xueting Wang, Rui Li, Jiebo Luo, Dongfang Liu, Yunbo Zhang

Keywords: One-shot learning, Egocentric video demonstration, Ensemble action prediction, Reinforcement learning

TL;DR: Robot learning via egocentric video

Abstract: Learning robot manipulation from egocentric video demonstrations is a challenging and promising direction for embodied intelligence, as it involves dynamic perspectives and uncertain environments. While existing methods have shown success in one-shot or few-shot learning from static videos, they are not applicable to egocentric video inputs, which significantly limits their scalability and real-world deployment. In this paper, we propose a novel coarse-to-fine directional manipulation learning framework that enables robots to acquire manipulation skills from a single egocentric video demonstration. Our method integrates an ensemble action prediction module for coarse action generation and a reinforcement learning-based refinement module for fine-grained, adaptive control. The ensemble module improves robustness by combining multiple diffusion policies, while the reinforcement module ensures accurate execution by refining motions based on real-time feedback. We evaluate our framework on three complex, multi-step manipulation tasks and demonstrate its superior performance over three state-of-the-art baselines in terms of both success rate and task robustness under one-shot egocentric settings.

Primary Area: applications to robotics, autonomy, planning

Submission Number: 1351