Compositional Interfaces for Compositional Generalization

Jelena Luketina, Jack Lanchantin, Sainbayar Sukhbaatar, Arthur Szlam

Keywords: compositional generalization, modular architectures, generalist agents

TL;DR: end-to-end modular architecture demonstrates compositional generalization wrt observation, action and instruction spaces

Abstract:

In this work, we study the effectiveness of a modular architecture for compositional generalization and transfer learning in the embodied agent setting. We develop an environment that allows us to independently vary perceptual modalities and action and task instructions, and use it to carefully analyze the agent's performance in these compositions. Our experiments demonstrate strong zero-shot performance
on held-out combinations of perception, action, and instruction spaces; as well as fast adaptation to new perceptual spaces without the loss of performance.

Submission Number: 40