Samples Are Useful? Not Always: denoising policy gradient updates using variance explained

Yannis Flet-Berliac, Philippe Preux

Keywords: reinforcement learning, policy gradient, sampling

TL;DR: SAUNA uses the fraction of variance explained (Vex) as a metric to filter the transitions used for policy gradient updates: such filtering improves the sampling prior for a better exploration of the environment and yields a better performance.

Abstract: Policy gradient algorithms in reinforcement learning optimize the policy directly and rely on efficiently sampling an environment. However, while most sampling procedures are based solely on sampling the agent's policy, other measures directly accessible through these algorithms could be used to improve sampling before each policy update. Following this line of thoughts, we propose the use of SAUNA, a method where transitions are rejected from the gradient updates if they do not meet a particular criterion, and kept otherwise. This criterion, the fraction of variance explained Vex, is a measure of the discrepancy between a model and actual samples. In this work, Vex is used to evaluate the impact each transition will have on learning: this criterion refines sampling and improves the policy gradient algorithm. In this paper: (a) We introduce and explore Vex, the criterion used for denoising policy gradient updates. (b) We conduct experiments across a variety of benchmark environments, including standard continuous control problems. Our results show better performance with SAUNA. (c) We investigate why Vex provides a reliable assessment for the selection of samples that will positively impact learning. (d) We show how this criterion can work as a dynamic tool to adjust the ratio between exploration and exploitation.

Code: https://github.com/iclr2020-submission/denoising-gradient-updates