Shared Autonomy for Robotic Manipulation with Language Corrections

Siddharth Karamcheti, Raj Palleti, Yuchen Cui, Percy Liang, Dorsa Sadigh

TL;DR: We introduce LILAC (Language-Informed Latent Actions with Corrections), a shared autonomy system for robotic manipulation that can handle streaming natural language corrections.

Abstract: Traditional end-to-end instruction following approaches for robotic manipulation are notoriously sample inefficient and lack adaptivity; for most single-turn methods, there is no way to provide additional language supervision to adapt robot behavior online – a property critical to deploying robots in collaborative, safety-critical environments. In this work, we present a method for incorporating language corrections, built on the insight that an initial instruction and subsequent corrections differ mainly in the amount of grounded context needed. To focus on manipulation domains where the sample efficiency of existing work is prohibitive, we incorporate our method into a shared autonomy system. Shared autonomy splits agency between the human and robot; rather than specifying a goal the robot needs to achieve alone, language informs the control space provided to the human. Splitting agency this way allows the robot to learn the coarse, high-level parts of a task, offloading more involved decisions – such as when to execute a grasp, or if a grasp is solid – to humans. Our user study on a Franka Emika Panda arm shows that our correction-aware system is sample-efficient and obtains significant gains over non-adaptive baselines.

Track: Non-Archival (will not appear in proceedings)