Hindsight Trust Region Policy Optimization

Hanbo Zhang, Site Bai, Xuguang Lan, Nanning Zheng

Keywords: Hindsight, Sparse Reward, Reinforcement Learning, Policy Gradients

TL;DR: This paper proposes an advanced policy optimization method with hindsight experience for sparse reward reinforcement learning.

Abstract: As reinforcement learning continues to drive machine intelligence beyond its conventional boundary, unsubstantial practices in sparse reward environment severely limit further applications in a broader range of advanced fields. Motivated by the demand for an effective deep reinforcement learning algorithm that accommodates sparse reward environment, this paper presents Hindsight Trust Region Policy Optimization (HTRPO), a method that efficiently utilizes interactions in sparse reward conditions to optimize policies within trust region and, in the meantime, maintains learning stability. Firstly, we theoretically adapt the TRPO objective function, in the form of the expected return of the policy, to the distribution of hindsight data generated from the alternative goals. Then, we apply Monte Carlo with importance sampling to estimate KL-divergence between two policies, taking the hindsight data as input. Under the condition that the distributions are sufficiently close, the KL-divergence is approximated by another f-divergence. Such approximation results in the decrease of variance and alleviates the instability during policy update. Experimental results on both discrete and continuous benchmark tasks demonstrate that HTRPO converges significantly faster than previous policy gradient methods. It achieves effective performances and high data-efficiency for training policies in sparse reward environments.

Code: https://github.com/HTRPOCODES/HTRPO

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:1907.12439/code)