Deep Neural Regression Collapse

Akshay Rangamani, Altay Unal

Keywords: Neural Collapse, Regression, Low Rank

Abstract: Neural Collapse is a phenomenon that helps identify sparse and low rank structures in deep classifiers. Recent work has extended the definition of neural collapse to regression problems, albeit only measuring the phenomenon at the last layer. In this work, we establish that neural regression collapse also occurs below the last layer through empirical measurements. We also outline parallels between the neural collapse in classifiers and regressors and show that the relevant measurements arise from an appropriate covariance decomposition. Identifying neural collapse in regression models can help us identify low rank structures in a wider class of deep networks.

Submission Number: 55