Transformers Are Adaptable Task Planners

Vidhi Jain, Yixin Lin, Eric Undersander, Yonatan Bisk, Akshara Rai

Keywords: Task Planning, Prompt, Preferences, Object-centric Representation

TL;DR: Transformer Task Planner (TTP) learns high-level pick and place actions from dishwasher loading demonstrations and adapts to unseen preferences using single prompt.

Abstract: Every home is different, and every person likes things done in their particular way. Therefore, home robots of the future need to both reason about the sequential nature of day-to-day tasks and generalize to user's preferences. To this end, we propose a Transformer Task Planner (TTP) that learns high-level actions from demonstrations by leveraging object attribute-based representations. TTP can be pre-trained on multiple preferences and shows generalization to unseen preferences using a single demonstration as a prompt in a simulated dishwasher loading task. Further, we demonstrate real-world dish rearrangement using TTP with a Franka Panda robotic arm, prompted using a single human demonstration.

Student First Author: yes

Website: https://sites.google.com/andrew.cmu.edu/ttp/home

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2207.02442/code)