Plan-Based Relaxed Reward Shaping for Goal-Directed Tasks

Ingmar Schubert, Ozgur S Oguz, Marc Toussaint

Keywords: reinforcement learning, reward shaping, plan-based reward shaping, robotics, robotic manipulation

Abstract: In high-dimensional state spaces, the usefulness of Reinforcement Learning (RL) is limited by the problem of exploration. This issue has been addressed using potential-based reward shaping (PB-RS) previously. In the present work, we introduce Final-Volume-Preserving Reward Shaping (FV-RS). FV-RS relaxes the strict optimality guarantees of PB-RS to a guarantee of preserved long-term behavior. Being less restrictive, FV-RS allows for reward shaping functions that are even better suited for improving the sample efficiency of RL algorithms. In particular, we consider settings in which the agent has access to an approximate plan. Here, we use examples of simulated robotic manipulation tasks to demonstrate that plan-based FV-RS can indeed significantly improve the sample efficiency of RL over plan-based PB-RS.

One-sentence Summary: We introduce Final-Volume-Preserving Reward Shaping, and show in a plan-based setting that it significantly increases the sample efficiency of reinforcement learning.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2107.06661/code)