Solving Non-Stationary Bandit Problems with an RNN and an Energy Minimization Loss

Michael Rotman, Lior Wolf

Keywords: Recurrent Neural Networks, Mutli Arm-Bandits

Abstract:

We consider a Multi-Armed Bandit problem in which the rewards are non-stationary and are dependent on past actions and potentially on past contexts. At the heart of our method, we employ a recurrent neural network, which models these sequences.
In order to balance between exploration and exploitation, we present an energy minimization term that prevents the neural network from becoming too confident in support of a certain action. This term provably limits the gap between the maximal and minimal probabilities assigned by the network. In a diverse set of experiments, we demonstrate that our method is at least as effective as methods suggested to solve the sub-problem of Rotting Bandits, can solve intuitive extensions of various benchmark problems, and is effective in a real-world recommendation system scenario.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: A very general non-stationary MAB framework and a new regularisation term that induces the required exploration

Reviewed Version (pdf): https://openreview.net/references/pdf?id=yQvyc7eB1c