On-Policy Policy Gradient Reinforcement Learning Without On-Policy Sampling

Nicholas Corrado, Josiah P. Hanna

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: reinforcement learning, on-policy, policy gradient, data collection

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We introduce a non-i.i.d., off-policy sampling method to produce data that more closely matches the expected on-policy data distribution than on-policy sampling can produce, thus improving the data efficiency of on-policy policy gradient algorithms.

Abstract:

On-policy reinforcement learning RL algorithms perform policy updates using i.i.d. trajectories collected by the current policy.
However, after observing only a finite number of trajectories, on-policy sampling may produce data that fails to match the expected on-policy data distribution.
This sampling error leads to noisy updates and data inefficient on-policy learning.
Recent work in the policy evaluation setting has shown that non-i.i.d., off-policy sampling can produce data with lower sampling error than on-policy sampling can produce~\citep{zhong2022robust}.
Motivated by this observation, we introduce an adaptive, off-policy sampling method to improve the data efficiency of on-policy policy gradient algorithms.
Our method, Proximal Robust On-Policy Sampling (PROPS) reduces sampling error by collecting data with a \textit{behavior policy} that increases the probability of sampling actions that are under-sampled with respect to the current policy.
Rather than discarding data from old policies -- as is commonly done in on-policy algorithms -- PROPS uses data collection to adjust the distribution of previously collected data to be approximately on-policy.
We empirically evaluate PROPS on both continuous-action MuJoCo benchmark tasks as well discrete-action tasks
and demonstrate that (1) PROPS decreases sampling error throughout training and (2) improves the data efficiency of on-policy policy gradient algorithms.
Our work improves the RL community’s understanding of a nuance in the on-policy vs off-policy dichotomy: on-policy learning requires on-policy data, not on-policy sampling.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4129