High Dimensional First Order Mini-Batch Algorithms on Quadratic Problems

Andrew Nicholas Cheng, Kiwon Lee, Courtney Paquette

Keywords: Optimization, Machine Learning, High-Dimensional Probability, Random Matrix Theory

Abstract: We analyze the dynamics of general mini-batch first order algorithms on the $\ell_2$ regularized least squares problem when the number of samples and dimensions are large. This includes stochastic gradient descent (SGD), stochastic Nesterov (convex/strongly convex), and stochastic momentum. In this setting, we show that the dynamics of these algorithms concentrate to a deterministic discrete Volterra equation $\Psi$ in the high-dimensional limit. In turn, we show that we can use $\Psi$ to capture the behaviour of general mini-batch first order algorithm under any quadratic statistics $\mathcal{R}: \mathbb{R}^d \to \mathbb{R}$, including but not limited to: training loss, excess risk for empirical risk minimization (in-distribution and generalization error.

Submission Number: 25