Toward Fair and Transparent Vision Transformers: Reproducing FairViT and Introducing FairDeiTA

TMLR Paper4284 Authors

Abstract: Vision Transformers (ViTs) have achieved state-of-the-art performance in image recognition but frequently inherit social biases from large-scale training data, raising concerns about fairness and transparency. FairViT was recently proposed to mitigate biases in ViTs through adaptive masking while preserving high accuracy with a distance-based loss. This study reproduces and evaluates FairViT’s claims on the CelebA dataset, focusing on accuracy and fairness metrics. Contrary to the original paper’s findings, our experiments reveal that FairViT does not outperform the baseline model in both performance and fairness. To enhance transparency, we apply interpretability techniques, including Gradient Attention Rollout (GAR) and local surrogate explanations (Ribeiro et al., 2016), providing deeper insight into the learned representations of FairViT. Our reproducibility study underscores the challenges of implementing and verifying fairness interventions in ViTs. Finally, we propose an adversarial debiasing (Zhang et al., 2018) component that improves fairness metrics while maintaining competitive accuracy, offering an alternative direction for fairness- focused ViT-based applications. We formulate this model as FairDeiTA.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Han_Zhao1

Submission Number: 4284