MimicPlay: Long-Horizon Imitation Learning by Watching Human PlayDownload PDF

Published: 30 Aug 2023, Last Modified: 13 Oct 2023CoRL 2023 OralReaders: Everyone
Keywords: Imitation Learning, Learning from Human, Long-Horizon Manipulation
TL;DR: We present MimicPlay, a novel imitation learning algorithm that leverages cost-effective human play data to learn long-horizon manipulation tasks in a sample-efficient manner.
Abstract: Imitation learning from human demonstrations is a promising paradigm for teaching robots manipulation skills in the real world. However, learning complex long-horizon tasks often requires an unattainable amount of demonstrations. To reduce the high data requirement, we resort to human play data - video sequences of people freely interacting with the environment using their hands. Even with different morphologies, we hypothesize that human play data contain rich and salient information about physical interactions that can readily facilitate robot policy learning. Motivated by this, we introduce a hierarchical learning framework named MimicPlay that learns latent plans from human play data to guide low-level visuomotor control trained on a small number of teleoperated demonstrations. With systematic evaluations of 14 long-horizon manipulation tasks in the real world, we show that MimicPlay outperforms state-of-the-art imitation learning methods in task success rate, generalization ability, and robustness to disturbances. Code and videos are available at https://mimic-play.github.io.
Student First Author: yes
Supplementary Material: zip
Instructions: I have read the instructions for authors (https://corl2023.org/instructions-for-authors/)
Website: https://mimic-play.github.io
Code: https://github.com/j96w/MimicPlay
Publication Agreement: pdf
11 Replies