Enhancing Deep Imbalanced Regression via Frobenius Norm Regularization

Ruizhi Pu, Yuanhang Yao, Song Tang, Charles Ling, Boyu Wang

Keywords: Deep Imbalanced Regression

Abstract:

Deep Imbalanced Regression (DIR) aims to train a deep neural network (DNN) model specified for the regression tasks from an imbalanced training distribution and generalize well on an unseen balanced testing distribution. While modern solutions have achieved significant progress in DIR, the performance of the samples still varies a lot across the different shots. For instance, the samples in the majority-shot always outperform the underrepresented (median and few-shot) samples, which motivates us to investigate whether we can leverage the well-trained majority-shot samples to help the other under-trained samples. Empirically, we observe that previous solutions in DIR often
produce ordinal feature Frobenius norms across the majority-shot samples and considerably lower training Mean-Absolute-Error (MAE).
Meanwhile, the underrepresented samples often violate the ordinality of the majority-shot Frobenius norms and exhibit a high training MAE.
As a result, this demonstrates that compared to the majority-shot samples, the underrepresented samples are still under-fitted during the training process. More importantly, we can identify the training performance through the lens of the ordinality of the Frobenius norm.
Motivated by this observation, we first analyze why the ordinality of the Frobenius norm can result in good training performance across the labels. Then, we introduce a feature regularization to encourage the feature Frobenius norms to be ordinal for all labels during the training process. Moreover, we propose a novel model training strategy that incorporates the knowledge from the well-trained majority samples to help the underrepresented samples. By training a linear model from the majority-shot samples to predict the feature Frobenius norm of underrepresented samples, we fine-tune the previously trained model to enhance the outcomes of underrepresented samples.
Extensive experiments over the real-world datasets also validate the effectiveness of our proposed method.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 2893