Open-Ended Learning Strategies for Learning Complex Locomotion Skills

Fangqin Zhou, Joaquin Vanschoren

Keywords: Reinforcement Learning, POET, ePOET, Locomotion Skills, Legged Robots, CPPN-NEAT

TL;DR: We adapt the Enhanced Paired Open-Ended Trailblazer (ePOET) approach to train complex agents to walk efficiently on complex three-dimensional terrains.

Abstract: Teaching robots to learn diverse locomotion skills under complex three-dimensional environmental settings via Reinforcement Learning (RL) is still challenging. It has been shown that training agents in simple settings before moving them on to complex settings improves the training process, but so far only in the context of relatively simple locomotion skills. In this work, we adapt the Enhanced Paired Open-Ended Trailblazer (ePOET) approach to train more complex agents to walk efficiently on complex three-dimensional terrains. First, to generate more rugged and diverse three-dimensional training terrains with increasing complexity, we extend the Compositional Pattern Producing Networks - Neuroevolution of Augmenting Topologies (CPPN-NEAT) approach and include randomized shapes. Second, we combine ePOET with Soft Actor-Critic off-policy optimization, yielding ePOET-SAC, to ensure that the agent could learn more diverse skills to solve more challenging tasks.

Contribution Process Agreement: Yes

Author Revision Details: Regarding the lack of discussion about how SAC helps the exploration of ePOET in our ePOET-SAC approach pointed out by the reviewers, although we had some discussion in our limitation section, to be more clear, we add further work to fine-tune the temperature parameter of SAC.

Poster Session Selection: Poster session #1 (12:00 UTC+1)

Process Comment: change from poster session 3 to session 1.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2206.06796/code)