Learning Transferable Policies By Inferring Agent Morphology

Brandon Trabucco, Mariano Phielipp, Glen Berseth

Keywords: Reinforcement Learning, Deep Learning, Transformers

TL;DR: We frame morphology as sequence modelling and propose a transformer policy that improves zero-shot generalization.

Abstract:

The prototypical approach to reinforcement learning involves training policies tailored to a particular agent from scratch for every new morphology.
Recent work aims to eliminate the re-training of policies by investigating whether a morphology-agnostic policy, trained on a diverse set of agents with similar task objectives, can be transferred to new agents with unseen morphologies without re-training. This is a challenging problem that required previous approaches to use hand-designed descriptions of the new agent's morphology. Instead of hand-designing this description, we propose a data-driven method that learns a representation of morphology directly from the reinforcement learning objective.
Ours is the first reinforcement learning algorithm that can train a policy to generalize to
new agent morphologies without requiring a description of the agent's morphology in advance. We evaluate our approach on a standard benchmark for agent-agnostic control, and improve over the state of the art in zero-shot generalization.
Importantly, our method attains good performance \textit{without} an explicit description of morphology.