Impulse Control Arbitration for A Dual System of Exploitation and Exploration

Changmin Yu, David Henry Mguni, Dong Li, Aivar Sootla, Jun Wang, Neil Burgess

Keywords: reinforcement learning. exploration-exploitation tradeoff, impulse control switching

TL;DR: We propose a plug-and-play framework with a learned impulse control switching mechanism for targeted arbitration between exploration and exploitation behaviour.

Abstract: Efficient reinforcement learning (RL) involves a trade-off between "exploitative" actions that maximise expected reward and ``explorative" ones that lead to the visitation of "novel" states. To encourage exploration, existing methods proposed methods such as injecting stochasticity into action selection, implicit regularisation, and additive synthetic reward. However, these techniques do not necessarily offer entirely systematic approaches making this trade-off. Here we introduce SElective Reinforcement EXploration (SEREX), a plug-and-play framework that casts the exploration-exploitation trade-off as a game between an RL agent--- exploiter, which purely exploits task-dependent rewards, and another RL agent--- switcher, which chooses at which states to activate a pure exploration policy that is trained to minimise system uncertainty and override Exploiter. Using a form of policies known as impulse control, switcher is able to determine the best set of states to switch to the exploration policy while Exploiter is free to execute its actions everywhere else. We prove that SEREX converges quickly and induces a natural schedule towards pure exploitation. Through extensive empirical studies in both discrete and continuous control benchmarks, we show that with minimal modification, SEREX can be readily combined with existing RL algorithms and yields significant improvement in performance.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Reinforcement Learning (eg, decision and control, planning, hierarchical RL, robotics)