Generalization for multiclass classification with overparameterized linear models

Vignesh Subramanian, Rahul Arya, Anant Sahai

Keywords: overparameterized, multiclass, classification, theory, generalization, interpolation, bi-level, Gaussian model

Abstract: Via an overparameterized linear model with Gaussian features, we provide conditions for good generalization for multiclass classification of minimum-norm interpolating solutions in an asymptotic setting where both the number of underlying features and the number of classes scale with the number of training points. The survival/contamination analysis framework for understanding the behavior of overparameterized learning problems is adapted to this setting, revealing that multiclass classification qualitatively behaves like binary classification in that, as long as there are not too many classes (made precise in the paper), it is possible to generalize well even in settings where regression tasks would not generalize. Besides various technical challenges, it turns out that the key difference from the binary classification setting is that there are relatively fewer training examples of each class in the multiclass setting as the number of classes increases, making the multiclass problem ``harder'' than the binary one.

TL;DR: Asymptotic region for multiclass classification in overparameterized Gaussian feature model learned by min-norm interpolation where the number of classes scales with the number of training points.