Interpreting Categorical Distributional Reinforcement Learning: An Implicit Risk-Sensitive Regularization Effect

Ke Sun, Yingnan Zhao, Enze Shi, Yafei Wang, Xiaodong Yan, Bei Jiang, Linglong Kong

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: distributional reinforcement learning, regularization, entropy

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We interpret distributional reinforcement learning from the perspective of regularization effect.

Abstract: The theoretical advantages of distributional reinforcement learning~(RL) over expectation-based RL remain elusive, despite its remarkable empirical performance. Starting from Categorical Distributional RL~(CDRL), our work attributes the potential superiority of distributional RL to its \textit{risk-sensitive entropy regularization}. This regularization stems from the additional return distribution information regardless of only its expectation via the return density function decomposition, a variant of the gross error model in robust statistics. Compared with maximum RL that explicitly optimizes the policy to encourage the exploration, we reveal that the resulting risk-sensitive entropy regularization of CDRL plays a different role as an augmented reward function. It implicitly optimizes policies for a risk-sensitive exploration towards true target return distributions, which helps to reduce the intrinsic uncertainty of the environment. Finally, extensive experiments verify the importance of this risk-sensitive regularization in distributional RL, as well as the mutual impacts of both explicit and implicit entropy regularization.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6360