Mitigating Spurious Correlations in Image Recognition Models using Performance-Based Feature Sampling
Track: tiny / short paper (up to 3 pages)
Keywords: Image Recognition, Spurious Correlations, Clustering, Resampling, Representation Learning
TL;DR: We propose a framework for correcting spurious correlations in image classifiers w.r.t a task-independent feature space. We use sparse clustering to flag candidate biasing features and decorrelate them from the target label via adaptive resampling.
Abstract: Existing methods for detecting and correcting spurious correlations in image recognition models often fail to identify biasing features due to incoherent groupings of biased images. There is also little exploration of targeted removal of spurious correlations in a low-dimensional feature space. To address these gaps, we propose Performance-Based Feature Sampling (PBFS), a systematic method for producing image recognition models that are de-biased w.r.t a given feature space. We introduce a method for producing coherent bias group proposals (i.e., semantically related images potentially sharing biasing feature(s)) and decorrelating biasing features from the target label using adaptive resampling. We demonstrate that our framework is able to correct for known spurious correlations, and through both established and our proposed metrics, we show that our method is able to de-bias image recognition models both w.r.t a high-dimensional feature space capturing complex representations and w.r.t low-dimensional feature spaces representing simple physical properties.
Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.
Format: Maybe: the presenting author will attend in person, contingent on other factors that still need to be determined (e.g., visa, funding).
Funding: Yes, the presenting author of this submission falls under ICLR’s funding aims, and funding would significantly impact their ability to attend the workshop in person.
Presenter: ~Aarav_Monga1
Submission Number: 47
Loading