Adaptive Regularized Newton Methods with Inexact Hessian

Alexander Shestakov, Andrei Semenov, Aleksandr Beznosikov, Alexander Gasnikov, Martin Takáč, Dmitry Kamzolov

Keywords: optimization, second-order methods

Abstract:

Newton’s Method is the most widespread high-order method, demanding only the gradient and the Hessian of the objective function. However, one of the main disadvantages of Newton's method is its lack of global convergence. There are various ways to deal with it: Cubic Regularization of the Newton Method, Damped Newton Method, Quasi-Newton Methods etc. Another disadvantage is the high iteration cost in modern high dimension problems. In this paper we derive the convergence of Regularization of Newton Method with Bregman divergences for convex, non-convex functions that can be applied with various inexact Hessians. By obtaining global convergence, complexity of iteration stays the same up to logarithmic factor. We show that our theory is validated by multiple experiments, showing competitive performance among other Newton Methods.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 12142