A Variational Perspective on High-Resolution ODEs

Hoomaan Maskan, Konstantinos C. Zygalakis, Alp Yurtsever

Keywords: Nesterov's accelerated gradient, gradient descent, Lyapunov function, gradient norm minimization, rate-matching, stochastic variance reduction, stochastic gradient descent, noisy gradient

TL;DR: Through a variational study on high-resolution ODEs, we propose better convergence rate of the gradient norm minimization for the NAG method, deeper study on the rate-matching technique, and an accelerated method for noisy gradients.

Abstract: We consider unconstrained minimization of smooth convex functions. We propose a novel variational perspective using forced Euler-Lagrange equation that allows for studying high-resolution ODEs. Through this, we obtain a faster convergence rate for gradient norm minimization using Nesterov's accelerated gradient method. Additionally, we show that Nesterov's method can be interpreted as a rate-matching discretization of an appropriately chosen high-resolution ODE. Finally, using the results from the new variational perspective, we propose a stochastic method for noisy gradients. Several numerical experiments compare and illustrate our stochastic algorithm with state of the art methods.

Submission Number: 8210