Closing the gap between SVRG and TD-SVRG with Gradient Splitting

Arsenii Mustafin, Alex Olshevsky, Ioannis Paschalidis

Abstract:

Temporal difference (TD) learning is a policy evaluation in reinforcement learning whose performance can be enhanced by variance reduction methods.
Recently, multiple works have sought to fuse TD learning with Stochastic Variance Reduced Gradient (SVRG) method to achieve a geometric rate of convergence.
However, the resulting convergence rate is significantly weaker than what is achieved by SVRG in the setting of convex optimization.
In this work we utilize a recent interpretation of TD-learning as the splitting of the gradient of an appropriately chosen function, thus simplifying the algorithm and fusing TD with SVRG. Our main result is a geometric convergence bound with predetermined learning rate of $1/8$, which is identical to the convergence bound available for SVRG in the convex setting. Our theoretical findings are supported by a set of experiments.

Submission Length: Regular submission (no more than 12 pages of main content)

Code: https://github.com/gaarsmu/SVRG_for_TD_learning

Assigned Action Editor: ~Michael_Bowling1

Submission Number: 2399