Inferential Induction: A Novel Framework for Bayesian Reinforcement Learning

Emilio Jorge, Hannes Eriksson, Christos Dimitrakakis, Debabrota Basu, Divya Grover

Keywords: reinforcement learning, bayesian reinforcement learning, policy evaluation

TL;DR: We describe a novel Bayesian framework, inferential induction, for correctly inferring value function distributions from data, which leads to a new family of Bayesian reinforcement learning algorithms.

Abstract: Bayesian Reinforcement Learning (BRL) offers a decision-theoretic solution to the reinforcement learning problem. While ''model-based'' BRL algorithms have focused either on maintaining a posterior distribution on models, BRL ''model-free'' methods try to estimate value function distributions but make strong implicit assumptions or approximations. We describe a novel Bayesian framework, \emph{inferential induction}, for correctly inferring value function distributions from data, which leads to a new family of BRL algorithms. We design an algorithm, Bayesian Backwards Induction (BBI), with this framework. We experimentally demonstrate that BBI is competitive with the state of the art. However, its advantage relative to existing BRL model-free methods is not as great as we have expected, particularly when the additional computational burden is taken into account.