Unifying Low Dimensional Observations in Deep Learning

Connall Garrod, Jonathan P. Keating

Keywords: deep learning, neural collapse, Hessian, spectral analysis, unconstrained feature model

TL;DR: This work shows that deep neural collapse explains the low-dimensional spectral structures of deep networks, with analytic formulas and empirical validation.

Abstract: Empirical studies have revealed low-dimensional structures in the eigenspectra of weights, Hessians, gradients, and feature vectors of deep networks, consistently observed across datasets and architectures in the overparameterized regime. In this work, we analyze deep unconstrained feature models (UFMs) to provide an analytic explanation of how these structures emerge at the layer-wise level, including the bulk–outlier Hessian spectrum and the alignment of gradient descent with the outlier eigenspace. We show that deep neural collapse underlies these phenomena, deriving explicit expressions for eigenvalues and eigenvectors of many deep learning matrices in terms of class feature means. Furthermore, we demonstrate that the full Hessian inherits its low-dimensional structure from the layer-wise Hessians, and empirically validate our theory in both UFMs and deep networks.

Primary Area: learning theory

Submission Number: 4868