Adam with model exponential moving average is effective for nonconvex optimization

Kwangjun Ahn, Ashok Cutkosky

Keywords: Adam, exponential moving average, EMA, nonconvex optimization.

Abstract: In this work, we offer a theoretical analysis of two modern optimization techniques for training large and complex models: (i) adaptive optimization algorithms, such as Adam, and (ii) the model exponential moving average (EMA). Specifically, we demonstrate that a clipped version of Adam with model EMA achieves the optimal convergence rates in various nonconvex optimization settings, both smooth and nonsmooth. Moreover, when the scale varies significantly across different coordinates, we demonstrate that the coordinate-wise adaptivity of Adam is provably advantageous. Notably, unlike previous analyses of Adam, our analysis crucially relies on its core elements---momentum and discounting factors---as well as model EMA, motivating their wide applications in practice.

Primary Area: Optimization (convex and non-convex, discrete, stochastic, robust)

Submission Number: 10882