Modularity through Attention: Efficient Training and Transfer of Language-Conditioned Policies for Robot ManipulationDownload PDF

Published: 10 Sept 2022, Last Modified: 12 Mar 2024CoRL 2022 PosterReaders: Everyone
Keywords: Language-Conditioned Learning, Attention, Imitation, Modularity
TL;DR: Efficient language-conditioned robot policy training and transfer through neural networks.
Abstract: Language-conditioned policies allow robots to interpret and execute human instructions. Learning such policies requires a substantial investment with regards to time and compute resources. Still, the resulting controllers are highly device-specific and cannot easily be transferred to a robot with different morphology, capability, appearance or dynamics. In this paper, we propose a sample-efficient approach for training language-conditioned manipulation policies that allows for rapid transfer across different types of robots. By introducing a novel method, namely Hierarchical Modularity, and adopting supervised attention across multiple sub-modules, we bridge the divide between modular and end-to-end learning and enable the reuse of functional building blocks. In both simulated and real world robot manipulation experiments, we demonstrate that our method outperforms the current state-of-the-art methods and can transfer policies across 4 different robots in a sample-efficient manner. Finally, we show that the functionality of learned sub-modules is maintained beyond the training process and can be used to introspect the robot decision-making process.
Student First Author: yes
Supplementary Material: zip
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 1 code implementation](
18 Replies