When should agents explore?

Miruna Pislar, David Szepesvari, Georg Ostrovski, Diana L Borsa, Tom Schaul

Keywords: exploration, mode-switching, reinforcement learning, Atari

TL;DR: A fresh look at the question of *when* to switch into exploration mode, and for how long.

Abstract: Exploration remains a central challenge for reinforcement learning (RL). Virtually all existing methods share the feature of a *monolithic* behaviour policy that changes only gradually (at best). In contrast, the exploratory behaviours of animals and humans exhibit a rich diversity, namely including forms of *switching* between modes. This paper presents an initial study of mode-switching, non-monolithic exploration for RL. We investigate different modes to switch between, at what timescales it makes sense to switch, and what signals make for good switching triggers. We also propose practical algorithmic components that make the switching mechanism adaptive and robust, which enables flexibility without an accompanying hyper-parameter-tuning burden. Finally, we report a promising initial study on Atari, using two-mode exploration and switching at sub-episodic time-scales.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Community Implementations: [ 6 code implementations](https://www.catalyzex.com/paper/arxiv:2108.11811/code)