A Closer Look at AUROC and AUPRC under Class Imbalance

Matthew B.A. McDermott, Haoran Zhang, Lasse Hyldig Hansen, Giovanni Angelotti, Jack Gallifant

Keywords: AUROC, AUPRC, Area under the receiver operating characteristic, Area under the precision recall curve, evaluation, fairness, disparities, bias

TL;DR: Contrary to popular belief, we show that AUPRC should *not* be generally preferred over AUROC in cases of class imbalance, and may even be a harmful metric as it can risk exacerbating algorithmic disparities.

Abstract: In machine learning (ML), a widespread claim is that the area under the precision-recall curve (AUPRC) is a superior metric for model comparison to the area under the receiver operating characteristic (AUROC) for tasks with class imbalance. This paper refutes this notion on two fronts. First, we theoretically characterize the behavior of AUROC and AUPRC in the presence of model mistakes, establishing clearly that AUPRC is not generally superior in cases of class imbalance. We further show that AUPRC can be a harmful metric as it can unduly favor model improvements in subpopulations with more frequent positive labels, heightening algorithmic disparities. Next, we empirically support our theory using experiments on both semi-synthetic and real-world fairness datasets. Prompted by these insights, we conduct a review of over 1.5 million scientific papers to understand the origin of this invalid claim, finding that it is often made without citation, misattributed to papers that do not argue this point, and aggressively over-generalized from source arguments. Our findings represent a dual contribution: a significant technical advancement in understanding the relationship between AUROC and AUPRC and a stark warning about unchecked assumptions in the ML community.

Primary Area: Fairness

Submission Number: 7584