Bayesian Reinforcement Learning for Single-Episode Missions in Partially Unknown Environments

Matthew Budd, Paul Duckworth, Nick Hawes, Bruno Lacerda

Keywords: Planning under Uncertainty, Gaussian Processes, Single-Episode Bayesian Reinforcement Learning

TL;DR: We propose a unified Bayes-optimal framework for single-mission robot planning in unknown environments, and evaluate on two realistic simulated environments.

Abstract: We consider planning for mobile robots conducting missions in real-world domains where a priori unknown dynamics affect the robot’s costs and transitions. We study single-episode missions where it is crucial that the robot appropriately trades off exploration and exploitation, such that the learning of the environment dynamics is just enough to effectively complete the mission. Thus, we propose modelling unknown dynamics using Gaussian processes, which provide a principled Bayesian framework for incorporating online observations made by the robot, and using them to predict the dynamics in unexplored areas. We then formulate the problem of mission planning in Markov decision processes under Gaussian process predictions as Bayesian model-based reinforcement learning. This allows us to employ solution techniques that plan more efficiently than previous Gaussian process planning methods are able to. We empirically evaluate the benefits of our formulation in an underwater autonomous vehicle navigation task and robot mission planning in a realistic simulation of a nuclear environment.

Student First Author: yes