XSkill: Cross Embodiment Skill DiscoveryDownload PDF

Published: 30 Aug 2023, Last Modified: 14 Oct 2023CoRL 2023 PosterReaders: Everyone
Keywords: Manipulation, Representation Learning, Cross-Embodiements
TL;DR: We present XSkill, an imitation learning framework that can discover, transfer and compose skills from cross-embodiment demonstration.
Abstract: Human demonstration videos are a widely available data source for robot learning and an intuitive user interface for expressing desired behavior. However, directly extracting reusable robot manipulation skills from unstructured human videos is challenging due to the big embodiment difference and unobserved action parameters. To bridge this embodiment gap, this paper introduces XSkill, an imitation learning framework that 1) discovers a cross-embodiment representation called skill prototypes purely from unlabeled human and robot manipulation videos, 2) transfers the skill representation to robot actions using conditional diffusion policy, and finally, 3) composes the learned skill to accomplish unseen tasks specified by a human prompt video. Our experiments in simulation and real-world environments show that the discovered skill prototypes facilitate both skill transfer and composition for unseen tasks, resulting in a more general and scalable imitation learning framework.
Student First Author: yes
Supplementary Material: zip
Instructions: I have read the instructions for authors (https://corl2023.org/instructions-for-authors/)
Video: https://xskill.cs.columbia.edu/
Website: https://xskill.cs.columbia.edu/
Code: https://xskill.cs.columbia.edu/
Publication Agreement: pdf
Poster Spotlight Video: mp4
11 Replies