Class-Weighted Evaluation Metrics for Imbalanced Data ClassificationDownload PDF

Published: 28 Jan 2022, Last Modified: 13 Feb 2023ICLR 2022 SubmittedReaders: Everyone
Keywords: Imbalanced data classification, Evaluation metrics, Log parsing, Sentiment analysis, URL classification
Abstract: Class distribution skews in imbalanced datasets may lead to models with prediction bias towards majority classes, making fair assessment of classifiers a challenging task. Metrics such as Balanced Accuracy are commonly used to evaluate a classifier’s prediction performance under such scenarios. However, these metrics fall short when classes vary in importance. In this paper, we propose a simple and general-purpose evaluation framework for imbalanced data classification that is sensitive to arbitrary skews in class cardinalities and importances. Experiments with several state-of-the-art classifiers tested on real-world datasets from three different domains show the effectiveness of our framework – not only in evaluating and ranking classifiers, but also training them.
One-sentence Summary: This paper presents a new evaluation framework for imbalanced data classification that can be used to train, evaluate, and rank models in a way that is sensitive to arbitrary skews in class cardinalities and importances.
Supplementary Material: zip
14 Replies

Loading