Implicit bias of Hessian Approximation in Regularized Randomized SR1 Method

ICLR 2026 Conference Submission17279 Authors

Keywords: Randomized SR1, Implicit bias, Hessian learning, Regularized Quasi-Newton methods

Abstract: Quasi-Newton methods have recently been shown to demonstrate dimension-independent convergence rate outperforming vanilla gradient descent (GD) in modern high-dimensional problems. By examining the spectrum of the Hessian approximation throughout the iterative process, we analyze a regularized quasi-Newton algorithm based on the standard randomized symmetric rank-one (SR1) update. The evolution of the spectrum reveals an implicit bias introduced by the Hessian learning, which promotes a preferential reduction of certain eigenvalues. This observation precisely captures the quality of Hessian approximation. Incorporating the implicit effect of Hessian update, we show that the regularized randomized SR1 method achieves a convergence rate of $\tilde{\mathcal O}\left(\frac{d_{\operatorname{eff}}^2}{k^2}\right)$ for standard self-concordant objective functions, where $d_{\operatorname{eff}}$ is the effective dimension of Hessian. In specific high-dimensional settings, which are common in practice, this method preserves convergence speeds comparable to accelerated gradient descent (AGD) while maintaining similar computational complexity per iteration. This work highlights the impact of implicit bias and offers a new perspective on the efficiency of quasi-Newton methods.

Primary Area: optimization

Submission Number: 17279