Talk Through It: End User Directed Manipulation Learning

Carl Winge, Karthik Desingh, Adam Imdieke, Bahaa Aldeeb, Dongyeop Kang

Keywords: Robotic manipulation, natural language, behavior cloning, VLM, end user

TL;DR: We present a framework that allows end users to instruct robots to complete tasks, and save the tasks as demonstrations for behavior cloning.

Abstract:

Training robots to perform a huge range of tasks in many different environments is immensely difficult. Instead, we propose selectively training robots based on end-user preferences.

Given a vision and language conditioned factory model that lets an end user instruct a robot to perform lower-level actions (e.g. ‘Move left’), we show that end users can collect demonstrations using language to train their home model for higher-level tasks specific to their needs (e.g. ‘Open the top drawer and put the block inside’). Our method results in a 13% improvement in task success rates compared to a baseline method.

We also explore the use of the large vision-language model (VLM), Bard, to automatically break down tasks into sequences of lower-level instructions, aiming to bypass end-user involvement. The VLM is unable to break tasks down to our lowest level, but does achieve good results breaking high-level tasks into mid-level skills. We have a supplemental video and additional results at talk-through-it.github.io.

Submission Number: 34