Perception-Driven Curiosity with Bayesian Surprise

Bernadette Bucher, Anton Arapin, Ramanan Sekar, Marc Badger, Feifei Duan, Oleh Rybkin, Kostas Daniilidis

Keywords: deep reinforcement learning, exploration, curiosity, variational methods, deep autoencoders

TL;DR: We introduce a method for computing an intrinsic reward for curiosity using metrics derived from sampling a latent variable model used to estimate dynamics.

Abstract: Intrinsic rewards in reinforcement learning provide a powerful algorithmic capability for agents to learn how to interact with their environment in a task-generic way. However, increased incentives for motivation can come at the cost of increased fragility to stochasticity. We introduce a method for computing an intrinsic reward for curiosity using metrics derived from sampling a latent variable model used to estimate dynamics. Ultimately, an estimate of the conditional probability of observed states is used as our intrinsic reward for curiosity. In our experiments, a video game agent uses our model to autonomously learn how to play Atari games using our curiosity reward in combination with extrinsic rewards from the game to achieve improved performance on games with sparse extrinsic rewards. When stochasticity is introduced in the environment, our method still demonstrates improved performance over the baseline.

Code: https://github.com/anonymous-iclr2020/surprise