Geometric properties of neural multivariate regression

George Andriopoulos, Bimarsha Adhikari, Zixuan Dong, Keith W. Ross

Keywords: neural collapse, regression, intrinsic dimension, generalization

Abstract: Neural multivariate regression underpins a wide range of domains such as control, robotics, finance, and meteorology, yet the geometry of its learned representations remains poorly characterized. While neural collapse has been shown to benefit generalization in classification, we find that analogous collapse in regression consistently degrades performance. To explain this contrast, we analyze models through the lens of intrinsic dimension. Across synthetic benchmarks and control tasks, we estimate the intrinsic dimension of last-layer features ($ID_H$) and compare it with that of the regression targets ($ID_Y$). Collapsed models exhibit $ID_H < ID_Y$, leading to over-compression and poor generalization, whereas non-collapsed models typically maintain $ID_H > ID_Y$. Moreover, the gap between IDH and IDY reliably predicts test error. From these observations, we identify two regimes—over-compressed and under-compressed—that determine when expanding or reducing feature dimensionality improves performance. Our results provide new geometric insights into neural regression and suggest practical strategies for enhancing generalization.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 8852