Adaptive Second-Order Stochastic Optimization
Keywords: MachinSecond-order optimization, stochastic Newton methods, stochastic quasi-Newton methods, adaptive graident descent, convergence analysis
Abstract: As a much possible way of improving first-order stochastic optimization ($\mathcal{FSO}$), the role of second-order information in stochastic optimization is receiving an increasing attention especially for solving the model with large-scale datasets in recent years, resulting in various second-order stochastic optimization ($\mathcal{SSO}$) methods, e.g., the stochastic Newton (SN) method, the stochastic quasi-Newton (SQN) method, etc. However, the question of how to set an appropriate update rule of the learning rate for SSO methods is still an extremely intractable task, and surprisingly there is quite less literature to tackle this issue. To bridge the gap between the SSO methods and the learning rate, this work develops a class of adaptive SSO methods from the perspective of adaptive gradient methods. Concretely, a general adaptive gradient (GAG) method with the quasi-hyperbolic momentum (QHM) strategy that encompasses Adam, AdaGrad, RMSProp, etc., as the special case of GAG, is incorporated into SN and SQN, respectively, which leads to two methods: SN-GAG and SQN-GAG. In addition, we establish a unified analysis for different adaptive SSO methods, covering their convergence behavior and computational complexity for different backgrounds, such as the strongly convex (SC) case and the Polyak-{\L}ojasiewicz (P{\L}) case, where, particularly, the latter is missing in current studies. Finally, numerical tests on different applications of machine learning demonstrate the superiority and the robustness of the resulting methods.
Primary Area: learning theory
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 29
Loading