AR2-D2: Training a Robot Without a Robot

Jiafei Duan, Yi Ru Wang, Mohit Shridhar, Dieter Fox, Ranjay Krishna

Keywords: Demonstration collection, Imitation learning, Augmented reality, Manipulating personalized objects, Dataset collection, Behavior Cloning

TL;DR: A novel framework for collecting trainable robot demonstrations without the need for a real robot.

Abstract:

Diligently gathered human demonstrations serve as the unsung heroes empowering the progression of robot learning.
Today, demonstrations are collected by training people to use specialized controllers, which (tele-)operate robots to manipulate a small number of objects. By contrast, we introduce AR2-D2: a system for collecting demonstrations which (1) does not require people with specialized training, (2) does not require any real robots during data collection, and therefore, (3) enables manipulation of diverse objects with a real robot. AR2-D2 is a framework in the form of an iOS app that people can use to record a video of themselves manipulating any object while simultaneously capturing essential data modalities for training a real robot. We show that data collected via our system enables the training of behavior cloning agents in manipulating real objects. Our experiments further show that training with our AR data is as effective as training with real-world robot demonstrations. Moreover, our user study indicates that users find AR2-D2 intuitive to use and require no training in contrast to four other frequently employed methods for collecting robot demonstrations.

Student First Author: yes

Instructions: I have read the instructions for authors (https://corl2023.org/instructions-for-authors/)

Video: https://www.youtube.com/watch?v=j-5iciBnXRc&embeds_referring_euri=http%3A%2F%2Far2d2.site%2F&source_ve_path=MjM4NTE&feature=emb_title

Website: www.ar2d2.site

Code: https://github.com/jiafei1224/AR2-D2_Utils