Investigating Spurious Correlations in Vision Models Using Counterfactual Images

Zichao Li

Keywords: Spurious Correlations, Counterfactual Images, Fairness, Robustness, Vision Models, Adversarial Perturbations

TL;DR: This paper investigates spurious correlations in vision models using counterfactual images, revealing biases and vulnerabilities while proposing strategies to improve fairness and robustness.

Abstract: Vision models often rely on spurious correlations, patterns in the data that are not intrinsic to the task but are nonetheless exploited by the model. These correlations can lead to biases, reduced robustness, and unfair predictions, particularly in sensitive domains like facial recognition and medical imaging. In this work, we systematically investigate spurious correlations using counterfactual image generation. By creating synthetic images with controlled variations in attributes such as texture, context, and demographics, we expose hidden biases in state-of-the-art vision models. Our experiments span multiple domains, including object recognition (ImageNet, COCO), face recognition (CelebA, FairFace), and medical imaging (CheXpert). We evaluate models for fairness, robustness, and generalization, revealing significant disparities across demographic groups and vulnerabilities to out-of-distribution samples and adversarial perturbations. Based on our findings, we propose actionable mitigation strategies, including data augmentation with counterfactuals, adversarial training, and fairness-aware regularization.

Submission Number: 6