Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor Critic

Tianying Ji, Yu Luo, Fuchun Sun, Xianyuan Zhan, Jianwei Dr. Zhang, Huazhe Xu

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Online RL, Off-policy Actor-Critic, Bellman operator, underexploitation issue

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We investigate the overlooked underestimationn issue in off-policy RL, propose the BEE operator, develop model-free and model-based algorithms based on it, and confirm the effectiveness through extensive simulations and real-world tasks.

Abstract:

Learning high-quality $Q$-value functions plays a key role in the success of many modern off-policy deep reinforcement learning (RL) algorithms.
Previous works focus on addressing the value overestimation issue, an outcome of adopting function approximators and off-policy learning.
Deviating from the common viewpoint, we observe that $Q$-values are indeed underestimated in the latter stage of the RL training process,
primarily related to the use of inferior actions from the current policy in Bellman updates as compared to the more optimal action samples in the replay buffer.
We hypothesize that this long-neglected phenomenon potentially hinders policy learning and reduces sample efficiency.
Our insight to address this issue is to incorporate sufficient exploitation of past successes while maintaining exploration optimism.
We propose the Blended Exploitation and Exploration (BEE) operator, a simple yet effective approach that updates $Q$-value using both historical best-performing actions and the current policy.
The instantiations of our method in both model-free and model-based settings outperform state-of-the-art methods in various continuous control tasks and achieve strong performance in failure-prone scenarios and real-world robot tasks

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5034