Exploring Uncertainty in Distributional Reinforcement Learning

Georgy Antonov, Peter Dayan

Keywords: Distributional reinforcement learning, exploration, uncertainty, Bayesian Q-learning

TL;DR: We propose a novel Bayesian distributional reinforcement learning algorithm and compare its exploration performance to a number of regular distributional RL algorithms.

Abstract: Epistemic uncertainty, which stems from what a learning algorithm does not know, is the natural signal for exploration. Capturing and exploiting epistemic uncertainty for efficient exploration is conceptually straightforward for model-based methods. However, it is computationally ruinous, prompting a search for model-free approaches. One of the most seminal and venerable such is Bayesian Q-learning, which maintains and updates an approximation to the distribution of the long run returns associated with state-action pairs. However, this approximation can be rather severe. Recent work on distributional reinforcement learning (DRL) provides many powerful methods for modelling return distributions which offer the prospect of improving upon Bayesian Q-learning's parametric scheme, but have not been fully investigated for their exploratory potential. Here, we examine the characteristics of a number of DRL algorithms in the context of exploration and propose a novel Bayesian analogue of the categorical temporal-difference algorithm. We show that this works well, converging appropriately to a close approximation to the true return distribution.

Submission Number: 122