Convergence of regularized agent-state based Q-learning in POMDPs

Amit Sinha, Matthieu Geist, Aditya Mahajan

Keywords: POMDPs, Q-learning, Regularization

TL;DR: This paper considers characterizing the convergence of Q-learning algorithms with regularization when using non-Markovian information structures (agent state) in a POMDP model.

Abstract: In this paper, we present a framework to understand the convergence of commonly used Q-learning reinforcement learning algorithms in practice. Two salient features of such algorithms are: (i) the Q-table is recursively updated using an agent state (such as the state of a recurrent neural network) which is not a belief state or an information state and (ii) policy regularization is often used to encourage exploration and stabilize the learning algorithm. We investigate the simplest form of such Q-learning algorithms which we call regularized agent-state based Q-learning (RASQL) and show that it converges under mild technical conditions to the fixed point of an appropriately defined regularized MDP, which depends on the stationary distribution induced by the behavioral policy. We also show that a similar analysis continues to work for a variant of RASQL that learns periodic policies. We present numerical examples to illustrate that the empirical convergence behavior matches with the proposed theoretical limit.

Confirmation: I understand that authors of each paper submitted to EWRL may be asked to review 2-3 other submissions to EWRL.

Serve As Reviewer: ~Amit_Sinha1

Track: Regular Track: unpublished work

Submission Number: 84