Emergence of Locomotion Behaviors in Rich Environments

Nicolas Heess, Dhruva Tirumala, Srinivasan Sriram, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, ziyu wang, S. M. Ali Eslami, Martin Riedmiller, David Silver

Abstract:

The reinforcement learning paradigm allows, in principle, for complex behaviours
to be learned directly from simple reward signals. In practice, however, it is
common to carefully hand-design the reward function to encourage a particular
solution, or to derive it from demonstration data. In this paper explore how a rich
environment can help to promote the learning of complex behavior. Specifically,
we train agents in diverse environmental contexts, and find that this encourages
the emergence of robust behaviours that perform well across a suite of tasks.
We demonstrate this principle for locomotion – behaviours that are known for
their sensitivity to the choice of reward. We train several simulated bodies on a
diverse set of challenging terrains and obstacles, using a simple reward function
based on forward progress. Using a novel scalable variant of policy gradient
reinforcement learning, our agents learn to run, jump, crouch and turn as required
by the environment without explicit reward-based guidance