On the Estimation Bias in Double Q-Learning

Zhizhou Ren, Guangxiang Zhu, Beining Han, Jianglun Chen, Chongjie Zhang

Keywords: Reinforcement learning, Q-learning, Estimation bias

Abstract: Double Q-learning is a classical method for reducing overestimation bias, which is caused by taking maximum estimated values in the Bellman operator. Its variants in the deep Q-learning paradigm have shown great promise in producing reliable value prediction and improving learning performance. However, as shown by prior work, double Q-learning is not fully unbiased and still suffers from underestimation bias. In this paper, we show that such underestimation bias may lead to multiple non-optimal fixed points under an approximated Bellman operation. To address the concerns of converging to non-optimal stationary solutions, we propose a simple and effective approach as a partial fix for underestimation bias in double Q-learning. This approach leverages real returns to bound the target value. We extensively evaluate the proposed method in the Atari benchmark tasks and demonstrate its significant improvement over baseline algorithms.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We prove that double Q-learning may have multiple non-optimal fixed points, and propose a simple approach to address this issue.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2109.14419/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=QSTaZzOH9I