Linear Stochastic Approximation: How Far Does Constant Step-Size and Iterate Averaging Go?

Chandrashekar Lakshminarayanan and Csaba Szepesva'ri,

Abstract: In this paper we study study constant step- size averaged linear stochastic approxima- tion. With an eye towards linear value es- timation in reinforcement learning, we ask whether for a given class of linear estimation problems i) a single universal constant step- size with ii) a C/t worst-case expected error with a class-dependent constant C > 0 can be guaranteed when the error is measured via an appropriate weighted squared norm. Such a result has recently been obtained in the con- text of linear least squares regression. We give examples that show that the answer to these questions in general is no. On the pos- itive side, we also characterize the instance dependent behavior of the error of the said algorithms, identify some conditions under which the answer to the above questions can be changed to the positive, and in partic- ular show instance-dependent error bounds of magnitude O(1/t) for the constant step- size iterate averaged versions of TD(0) and a novel variant of GTD, where the stepsize is chosen independently of the value estimation instance. Computer simulations are used to illustrate and complement the theory.