Following Ancestral Footsteps: Co-Designing Agent Morphology and Behaviour with Self-Imitation Learning

Sergio Hernández-Gutiérrez, Ville Kyrki, Kevin Sebastian Luck

Keywords: Co-Adaptation, Reinforcement Learning, Self-Imitation Learning, Co-Design

TL;DR: When co-adapting agent morphology and behaviour it helps to imitate the behaviour of previous morphologies!

Abstract:

In this paper we consider the problem of co-adapting the body and behaviour of agents, a long-standing research problem in the community of evolutionary robotics.
Previous work has largely focused on the development of methods exploiting massive parallelization of agent evaluations with large population sizes, a paradigm which is not applicable to the real world.
More recent data-efficient approaches utilizing reinforcement learning can suffer from distributional shifts to transition dynamics as well as to states and action spaces when experiencing new body morphologies.
In this work, we propose a new co-adaptation method combining reinforcement learning and State-Aligned Self-Imitation Learning.
We show that the integration of a self-imitation signal improves data-efficiency, behavioural recovery for unseen designs and performance convergence.

Submission Number: 4