Hessian Spectrum is Constant Across Minimizers in Regularized Deep Scalar Factorization

Anıl Kamber, Rahul Parhi

Keywords: Deep Matrix Factorization, Flatness, Loss Landscape Analysis, Regularization

TL;DR: Hessian Spectrum is Constant Across Minimizers in Regularized Deep Scalar Factorization

Abstract: We characterize the full Hessian spectrum of $\ell^2$-regularized deep scalar factorization problems across all minimizers. We prove that the spectrum is constant across all minimizers and, in particular, that the maximum eigenvalue depends on the depth, the (shared) magnitude of optimal layers, and the regularization parameter. The limit of the regularization parameter to zero recovers the unregularized case for flat minima. To the best of our knowledge, our results offer the first complete characterization of Hessian spectrum across minimizers in deep-factorization-type problems.

Submission Number: 84