Exploration using Distributional RL and UCB

Borislav Mavrin, Hengshuai Yao, Linglong Kong, ShangtongZhang

Abstract:

We establish the relation between Distributional RL and the Upper Confidence Bound (UCB) approach to exploration.
In this paper we show that the density of the Q function estimated by Distributional RL can be successfully used for the estimation of UCB. This approach does not require counting and, therefore, generalizes well to the Deep RL. We also point to the asymmetry of the empirical densities estimated by the Distributional RL algorithms like QR-DQN. This observation leads to the reexamination of the variance's performance in the UCB type approach to exploration. We introduce truncated variance as an alternative estimator of the UCB and a novel algorithm based on it. We empirically show that newly introduced algorithm achieves better performance in multi-armed bandits setting. Finally, we extend this approach to high-dimensional setting and test it on the Atari 2600 games. New approach achieves better performance compared to QR-DQN in 26 of games, 13 ties out of 49 games.

Keywords: Distributional RL, UCB, exploration, Atari 2600, multi-armed bandits

TL;DR: Exploration using Distributional RL and truncagted variance.