Build generally reusable agent-environment interaction models

Jun Jin, Hongming Zhang, Jun Luo

Keywords: reinforcement learning, embedded agency, embodied cognition, embodied AI, pre-training, world models, successor features

TL;DR: Downstream task learning by projecting on a pre-trained embodied set structure can handle heterogeneous task settings.

Abstract: This paper tackles the problem of how to pre-train a model and make it generally reusable backbones for downstream task learning. In pre-training, we propose a method that builds an agent-environment interaction model by learning domain invariant successor features from the agent's vast experiences covering various tasks, then discretize them into behavior prototypes which result in an embodied set structure. To make the model generally reusable for downstream task learning, we propose (1) embodied feature projection that retains previous knowledge by projecting the new task's observation-action pair to the embodied set structure and (2) projected Bellman updates which add learning plasticity for the new task setting. We provide preliminary results that show downstream task learning based on a pre-trained embodied set structure can handle unseen changes in task objectives, environmental dynamics and sensor modalities.