Risk Phase Transitions in Spiked Regression: Alignment Driven Benign and Catastrophic Overfitting

Jiping Li, Rishi Sonthalia

Keywords: Generalization, Random Matrix Theory, Spiked Covariance, Benign/Tempered/Catastrophic Overfitting

Abstract:

This paper analyzes the generalization error of minimum-norm interpolating solutions in linear regression using spiked covariance data models. The paper characterizes how varying spike strengths and target-spike alignments affect risk, especially in overparameterized settings. The study presents an exact expression for the generalization error, leading to a comprehensive classification of benign, tempered, and catastrophic overfitting regimes based on spike strength, the aspect ratio $c=d/n$ (particularly as $c \to \infty$), and target alignment. Notably, in well-specified aligned problems, increasing spike strength can surprisingly induce catastrophic overfitting before achieving benign overfitting. The paper also reveals that target-spike alignment is not always advantageous, identifying specific, sometimes counterintuitive, conditions for its benefit or detriment. Alignment with the spike being detrimental is empirically demonstrated to persist in nonlinear models.

Student Paper: Yes

Submission Number: 54