A Hitchhiker's Guide to Momentum

Fabian Pedregosa

Published: 02 May 2023, Last Modified: 02 May 2023Blogposts @ ICLR 2023Readers: Everyone
Keywords: optimization, momentum, heavy ball
Abstract: Polyak momentum is one of the most iconic methods in optimization. Despite it's simplicit, it features rich dynamics that depend both on the step-size and momentum parameter. In this blog post we identify the different regions of the parameter space and discuss their convergence properties using the theory of Chebyshev polynomials.
Blogpost Url: https://iclr-blogposts.github.io/2023/blog/2023/hitchhikers-momentum/
ICLR Papers: https://openreview.net/forum?id=rJTutzbA-, https://openreview.net/forum?id=r1gixp4FPH, https://openreview.net/forum?id=L7WD8ZdscQ5
ID Of The Authors Of The ICLR Paper: ~Fabian_Pedregosa1
Conflict Of Interest: No
5 Replies