Go to ICLR 2025 Conference homepageSepNorm: Generalization of Lion and Normalised Gradient Methods
In this paper, we investigate the novel optimizer Lion (Evolved Sign Momentum), which demonstrates superior performance compared to the well-established Adam in a wide range of tasks. Lion is a combination of Sign Gradient Descent (SignGD) and momentum, utilizing a fixed step size and adjusting the gradient direction via a sign operation. Despite its promising results, Lion currently lacks comprehensive theoretical justification. We also discuss Normalized Gradient Descent methods, characterized by a fixed step size, which predate Lion. We show that both Lion and NormGD have notable disadvantages, and to address these issues, we propose a new method SepNorm, which normalizes gradients across different parameter groups. SepNorm generalizes both Lion and NormGD, offering a more adaptable and stable optimization approach. Our theoretical analysis on quadratic functions reveals mechanisms of convergence behind the methods and allows us to formulate implicit bias criteria for them. Additionally, we introduce OrtSepNorm, an extension of SepNorm that makes update direction orthogonal to the weights, and we demonstrate that OrtSepNorm converges to a fixed weight norm, thereby making the training process more stable. Empirical evaluations reveal that SepNorm and OrtSepNorm outperform both Lion and Adam in a range of computer vision (CV) and natural language processing (NLP) tasks.