Reward Constrained Policy Optimization

Chen Tessler, Daniel J. Mankowitz, Shie Mannor

Abstract: Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work we present a novel multi-timescale approach for constrained policy optimization, called `Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.

Keywords: reinforcement learning, markov decision process, constrained markov decision process, deep learning

TL;DR: For complex constraints in which it is not easy to estimate the gradient, we use the discounted penalty as a guiding signal. We prove that under certain assumptions it converges to a feasible solution.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/reward-constrained-policy-optimization/code)