Beyond Spectra: Eigenvector Overlaps in Loss Geometry

ICLR 2026 Conference Submission13989 Authors

Keywords: hessian, overlap, eigenvector, geometry, ridge regression, noise, free probability, algorithms, CIFAR, high dimensional statistics, generalization, covariate shift, double descent, multiple descent, random matrix theory

TL;DR: Loss geometry depends not only on train and test Hessian spectra but also on alignment of eigenspaces; we derive a universal fluctuation law, explain covariate shift and multiple descent, and develop scalable estimators for overlaps in large NNs.

Abstract: Local loss geometry in machine learning is fundamentally a two-operator concept. When only a single loss is considered, geometry is fully summarized by the Hessian spectrum; in practice, however, both training and test losses are relevant, and the resulting geometry depends on their spectra together with the alignment of their eigenspaces. We first establish general foundations for two-loss geometry by formulating a universal local fluctuation law, showing that the expected test-loss increment under small training perturbations is a trace that combines train and test spectral data with a critical additional factor quantifying eigenspace overlap, and by proving a novel transfer law that describes how overlaps transform in response to noise. As a solvable analytical model, we next apply these laws to ridge regression with arbitrary covariate shift, where operator-valued free probability yields asymptotically exact overlap decompositions that reveal overlaps as the natural quantities specifying shift and that resolve the puzzle of multiple descent: peaks are controlled by eigenspace (mis-)alignment rather than by Hessian ill-conditioning alone. Finally, for empirical validation and scalability, we confirm the fluctuation law in multilayer perceptrons, develop novel algorithms based on subspace iteration and kernel polynomial methods to estimate overlap functionals, and apply them to a ResNet-20 trained on CIFAR10, showing that class imbalance reshapes train–test loss geometry via induced misalignment. Together, these results establish overlaps as the critical missing ingredient for understanding local loss geometry, providing both theoretical foundations and scalable estimators for analyzing generalization in modern neural networks.

Primary Area: learning theory

Submission Number: 13989