LOCO: Adaptive exploration in reinforcement learning via local estimation of contraction coefficientsDownload PDF

Mar 09, 2021 (edited Apr 15, 2021)ICLR 2021 Workshop SSL-RL Blind SubmissionReaders: Everyone
  • Keywords: exploration, contraction coefficients, Markov Chain, mixing time
  • TL;DR: Global to Local Complexity of Random Exploration Through Contraction Coefficients.
  • Abstract: We offer a novel approach to balance exploration and exploitation in reinforcement learning (RL). To do so, we characterize an environment’s exploration difficulty via the Second Largest Eigenvalue Modulus (SLEM) of the Markov chain induced by uniform stochastic behaviour. Specifically, we investigate the connection of state-space coverage with the SLEM of this Markov chain and use the theory of contraction coefficients to derive estimates of this eigenvalue of interest. Furthermore, we introduce a method for estimating the contraction coefficients on a local level and leverage it to design a novel exploration algorithm. We evaluate our algorithm on a series of GridWorld tasks of varying sizes and complexity.
0 Replies