Steepest Descent in the Modular Norm
Keywords: Adam, Shampoo, Prodigy, optimizers, optimization, steepest descent, norms, modular norm, spectral norm
TL;DR: We cast optimizers like Adam and Shampoo as steepest descent methods under different norms. Generalizing this idea opens up a new design space for training algorithms
Abstract: An old idea in optimization theory says that since the gradient is a dual vector it may not be subtracted from the weights without first being mapped to the primal space where the weights live. We take this idea seriously in this paper and construct such a gradient mapping for neural networks. We call our mapping modular dualization, and show how it may be used to derive steepest descent optimization methods in an architecture-aware norm called the modular norm. We prove that the Adam and Shampoo optimizers (without exponential moving averages) emerge as special cases of modular dualization. We go on to develop new special-purpose dualize functions for linear, embedding and convolution layers. And we contribute a novel, GPU-friendly algorithm for dualizing under the spectral norm. We hope that the technique of modular dualization might enable a next generation of fast and scalable optimizers that are well-matched to general neural architectures.
Primary Area: optimization
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 13003
Loading