Suppressing Overestimation in Q-Learning through Adversarial Behaviors
Keywords: Markav decision process, reinforcement Learning, Q-learning, double Q-learning, overestimation, adversarial reinforcement learning, two-player zero-sum game
Abstract: The goal of this paper is to propose a new Q-learning algorithm with a dummy adversarial player, which is called dummy adversarial Q-learning (DAQ), that can effectively control the overestimation bias in standard Q-learning. With the dummy player, the learning can be formulated as a two-player zero-sum game. The proposed DAQ unifies several Q-learning variations to control overestimation, such as Maxmin Q-learning and Minmax Q-learning (proposed in this paper) in a single framework.
The proposed DAQ is a simple but effective way to suppress the overestimation bias from subtle rewards and can be easily applied to off-the-shelf reinforcement learning algorithms.
A finite-time convergence of DAQ is analyzed from an integrated perspective by adapting an adversarial Q-learning. The performance of the suggested DAQ is empirically demonstrated under various MDP environments.
Primary Area: reinforcement learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3412
Loading