Efficient Exploration through Bayesian Deep Q-Networks

Kamyar Azizzadenesheli, Emma Brunskill, Animashree Anandkumar

Abstract: We propose Bayesian Deep Q-Network (BDQN), a practical Thompson sampling based Reinforcement Learning (RL) Algorithm. Thompson sampling allows for targeted exploration in high dimensions through posterior sampling but is usually computationally expensive. We address this limitation by introducing uncertainty only at the output layer of the network through a Bayesian Linear Regression (BLR) model, which can be trained with fast closed-form updates and its samples can be drawn efficiently through the Gaussian distribution. We apply our method to a wide range of Atari Arcade Learning Environments. Since BDQN carries out more efficient exploration, it is able to reach higher rewards substantially faster than a key baseline, DDQN.

TL;DR: Using Bayesian regression to estimate the posterior over Q-functions and deploy Thompson Sampling as a targeted exploration strategy with efficient trade-off the exploration and exploitation

Keywords: Deep RL, Thompson Sampling, Posterior update

Code: [ kazizzad/BDQN-MxNet-Gluon](https://github.com/kazizzad/BDQN-MxNet-Gluon)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/efficient-exploration-through-bayesian-deep-q/code)