A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD

Ruinan Jin, Xiao Li, Yaoliang Yu, Baoxiang Wang

TL;DR: A theoretical paper on the convergence of Adam

Abstract:

Adaptive moment estimation (Adam) is a cornerstone optimization algorithm in deep learning, widely recognized for its flexibility with adaptive learning rates and efficiency in handling large-scale data. However, despite its practical success, the theoretical understanding of Adam's convergence has been constrained by stringent assumptions, such as almost surely bounded stochastic gradients or uniformly bounded gradients, which are more restrictive than those typically required for analyzing stochastic gradient descent (SGD).

In this paper, we introduce a novel and comprehensive framework for analyzing the convergence properties of Adam. This framework offers a versatile approach to establishing Adam's convergence. Specifically, we prove that Adam achieves asymptotic (last iterate sense) convergence in both the almost sure sense and the \(L_1\) sense under the relaxed assumptions typically used for SGD, namely \(L\)-smoothness and the ABC inequality. Meanwhile, under the same assumptions, we show that Adam attains non-asymptotic sample complexity bounds similar to those of SGD.

Lay Summary: Adam is one of the most popular optimization methods used to train deep learning models. It works well in practice because it can automatically adjust how fast it learns during training. However, until now, understanding exactly when and why Adam works has required very strong and often unrealistic mathematical assumptions. In this paper, we present a new theoretical framework that shows Adam can succeed under much more relaxed and practical conditions—similar to those needed to analyze the more basic algorithm SGD (stochastic gradient descent). Our results show that Adam not only performs well in practice but also has strong theoretical guarantees, helping bridge the gap between its empirical success and formal understanding. This work may also help researchers analyze other similar optimization methods more easily.

Link To Code: no code

Primary Area: Optimization->Stochastic

Keywords: Adam, ABC Inequality, Sample Complexity, Almost Sure Convergence, L_1 Convergence

Submission Number: 1314