Human-in-the-Loop Task and Motion Planning for Imitation Learning

Ajay Mandlekar, Caelan Reed Garrett, Danfei Xu, Dieter Fox

Keywords: Imitation Learning, Task and Motion Planning, Teleoperation

TL;DR: We present Human-in-the-Loop Task and Motion Planning (HITL-TAMP), a novel system that selectively gives and takes control to and from a human teleoperator, enabling more efficient imitation learning.

Abstract: Imitation learning from human demonstrations can teach robots complex manipulation skills, but is time-consuming and labor intensive. In contrast, Task and Motion Planning (TAMP) systems are automated and excel at solving long-horizon tasks, but they are difficult to apply to contact-rich tasks. In this paper, we present Human-in-the-Loop Task and Motion Planning (HITL-TAMP), a novel system that leverages the benefits of both approaches. The system employs a TAMP-gated control mechanism, which selectively gives and takes control to and from a human teleoperator. This enables the human teleoperator to manage a fleet of robots, maximizing data collection efficiency. The collected human data is then combined with an imitation learning framework to train a TAMP-gated policy, leading to superior performance compared to training on full task demonstrations. We compared HITL-TAMP to a conventional teleoperation system --- users gathered more than 3x the number of demos given the same time budget. Furthermore, proficient agents (75\%+ success) could be trained from just 10 minutes of non-expert teleoperation data. Finally, we collected 2.1K demos with HITL-TAMP across 12 contact-rich, long-horizon tasks and show that the system often produces near-perfect agents. Videos and additional results at https://hitltamp.github.io .

Student First Author: no

Instructions: I have read the instructions for authors (https://corl2023.org/instructions-for-authors/)

Website: https://hitltamp.github.io