Mitigating Biases in Blackbox Feature Extractors for Image Classification Tasks

Abhipsa Basu, Saswat Subhajyoti Mallick, Venkatesh Babu Radhakrishnan

Keywords: bias;fairness;spurious correlation

TL;DR: Analyses the propagation of biases in presence of blackbox feature extractors and suggests a simple migitation strategy for image classification

Abstract: In image classification, it is common to utilize a pretrained model to extract meaningful features of the input images, and then to train a classifier on top of it to make predictions for any downstream task. Trained on enormous amounts of data, these models have been shown to contain harmful biases which can hurt their performance when adapted for a downstream classification task. Further, very often they may be blackbox, either due to scale, or because of unavailability of model weights or architecture. Thus, during a downstream task, we cannot debias such models by updating the weights of the feature encoder, as only the classifier can be finetuned. In this regard, we investigate the suitability of some existing debiasing techniques and thereby motivate the need for more focused research towards this problem setting. Furthermore, we propose a simple method consisting of a clustering-based adaptive margin loss with a blackbox feature encoder, with no knowledge of the bias attribute. Our experiments demonstrate the effectiveness of our method across multiple benchmarks.

Primary Area: Fairness

Submission Number: 14668