Understanding Overfitting in Reweighting Algorithms for Worst-group Performance

Runtian Zhai, Chen Dan, J Zico Kolter, Pradeep Kumar Ravikumar

Keywords: Reweighting algorithms, Worst-group performance, Implicit bias, Fairness

Abstract: Prior work has proposed various reweighting algorithms to improve the worst-group performance of machine learning models for fairness. However, Sagawa et al. (2020) empirically found that these algorithms overfit easily in practice under the overparameterized setting, where the number of model parameters is much greater than the number of samples. In this work, we provide a theoretical backing to the empirical results above, and prove the pessimistic result that reweighting algorithms always overfit. Specifically we prove that with reweighting, an overparameterized model always converges to the same ERM interpolator that fits all training samples, and consequently its worst-group test performance will drop to the same level as ERM in the long run. That is, we cannot hope for reweighting algorithms to converge to a different interpolator than ERM with potentially better worst-group performance. Then, we analyze whether adding regularization helps fix the issue, and we prove that for regularization to work, it must be large enough to prevent the model from achieving small training error. Our results suggest that large regularization (or early stopping) and data augmentation are necessary for reweighting algorithms to achieve high worst-group test performance.

One-sentence Summary: We prove the pessimistic result that reweighting algorithms always overfit, and a useful regularization must be large enough to lower the training performance.