Improving Sample Efficiency in Off-policy RL with Low-dimensional Policy Representation

Min Zhang, Jianye HAO, Hongyao Tang, YAN ZHENG

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Reinforcement Learning, Off-policy, Representation Learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Off-policy Reinforcement Learning (RL) is fundamental to realizing intelligent decision-making agents by trial and error.
The most notorious issue of off-policy RL is known as Deadly Triad, i.e., Bootstrapping, Function Approximation, and Off-policy Learning. Despite recent advances in bootstrapping algorithms with better bias control, improvements in the latter two factors are relatively less studied. In this paper, we propose an efficient and general off-policy RL algorithm based on the low-dimensional policy representation.
Orthogonal to better bootstrapping, our improvement is two-fold. On the one hand, the policy representation serves as an additional input to the value function, allowing it to offer preferable function approximation with less interference and better generalization. On the other hand, the policy representation empowers off-policy RL methods to perform off-policy learning in a more sufficient manner. Specifically, we perform additional value learning for proximal historical policies along the learning process. This drives the value generalization from learned policies and in turn, leads to more efficient learning. We evaluate our algorithms on continuous control tasks and the empirical results demonstrate consistent improvements in terms of efficiency and stability.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4452