Compositional Interfaces for Compositional Generalization

Published: 20 Jun 2023, Last Modified: 16 Jul 2023ES-FoMO 2023 PosterEveryoneRevisionsBibTeX
Keywords: compositional generalization, modular architectures, generalist agents
TL;DR: end-to-end modular architecture demonstrates compositional generalization wrt observation, action and instruction spaces
Abstract: In this work, we study the effectiveness of a modular architecture for compositional generalization and transfer learning in the embodied agent setting. We develop an environment that allows us to independently vary perceptual modalities and action and task instructions, and use it to carefully analyze the agent's performance in these compositions. Our experiments demonstrate strong zero-shot performance on held-out combinations of perception, action, and instruction spaces; as well as fast adaptation to new perceptual spaces without the loss of performance.
Submission Number: 40