Bayesian Deep Q-Learning via Sequential Monte Carlo

Pascal Van der Vaart, Matthijs T. J. Spaan, Neil Yorke-Smith

Keywords: Bayesian deep learning, reinforcement learning, bayesian neural networks, ensembles

TL;DR: Applying sequential Monte Carlo to approximate the posteriors in bayesian neural networks for uncertainty quantification in reinforcement learning

Abstract: Exploration in reinforcement learning remains a difficult challenge. Recently, ensembles with randomized prior functions have been popularized to quantify uncertainty in the value model, in order to drive exploration with success. However these ensembles have no theoretical guarantee to resemble the actual posterior. In this work, we view training ensembles from the perspective of sequential Monte Carlo, and propose an algorithm that exploits both the practical flexibility of ensembles and theory of the Bayesian paradigm. We incorporate this method into a standard DQN agent and experimentally show improved exploration capabilities over a regular ensemble.