Mitigating Spurious Correlations in Image Recognition Models using Performance-Based Feature Sampling

Aarav Monga, Rajakrishnan Somou, Sonia Zhang, Antonio Ortega

Track: tiny / short paper (up to 3 pages)

Keywords: Image Recognition, Spurious Correlations, Clustering, Resampling, Representation Learning

TL;DR: We propose a framework for correcting spurious correlations in image classifiers w.r.t a task-independent feature space. We use sparse clustering to flag candidate biasing features and decorrelate them from the target label via adaptive resampling.

Abstract: Existing methods for detecting and correcting spurious correlations in image recognition models often fail to identify biasing features due to incoherent groupings of biased images. There is also little exploration of targeted removal of spurious correlations in a low-dimensional feature space. To address these gaps, we propose Performance-Based Feature Sampling (PBFS), a systematic method for producing image recognition models that are debiased w.r.t. a given feature space. We introduce a method for producing coherent bias group proposals (i.e., semantically related images potentially sharing biasing feature(s)) and decorrelating biasing features from the target label using adaptive resampling. We demonstrate that our framework is able to correct for known spurious correlations, and through both established and our proposed metrics, we show that our method is able to de-bias image recognition models both w.r.t a high-dimensional feature space capturing complex representations and w.r.t. low-dimensional feature spaces representing simple physical properties.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 47