An Analytical Model for Overparameterized Learning Under Class Imbalance

Eliav Mor, Yair Carmon

Abstract: We study class-imbalanced linear classification in a high-dimensional Gaussian mixture model. We develop a tight, closed form approximation for the test error of several practical learning methods, including logit adjustment and class dependent temperature. Our approximation allows us to analytically tune and compare these methods, highlighting how and when they overcome the pitfalls of standard cross-entropy minimization. We test our theoretical findings on simulated data and imbalanced CIFAR10, MNIST and FashionMNIST datasets.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

1. As requested by reviewer JvdP, we moved eq.22 (now eq.23) from appendix A.2 to the main paper (under section 3.1) to clarify the relation between w_y and the kernel matrix K.
2. Added an acknowledgements section.
3. Update footnote 1: "The "overparameterized regime" characterizes a setting where the number of parameters in a model significantly exceeds the size of the training set, allowing the model to interpolate the training set."
4. Updated Figure 1 legend.

Assigned Action Editor: ~Chris_J_Maddison1

Submission Number: 3659