Mitigating Demographic Bias in Vision Transformers via Attention-Guided Fair Representation Learning

Zichao Li, Zong Ke

Keywords: fairness, vision transformers, attention mechanisms, algorithmic bias, demographic diversity, deep learning, computer vision

TL;DR: We propose first attention-head pruning method for ViTs reduces racial bias by 41% and geographic bias by 7.1% while maintaining 83.5% accuracy, achieving new SOTA fairness-accuracy tradeoffs.

Abstract: We propose a novel attention-based debiasing framework for Vision Transformers (ViTs) that identifies and mitigates demographic biases through targeted head pruning and adaptive reweighting. Our method achieves state-of-the-art fairness-accuracy trade-offs on three benchmarks (FairFace, GeoDE, PPB), reducing racial bias by 40.9% (Fitzpatrick Type VI) and geographic bias by 7.1% while maintaining 83.5% accuracy on majority groups. Comprehensive experiments demonstrate consistent improvements over adversarial debiasing (+3.9-8.7% for marginalized groups) and token-level approaches (+2.8%), with <1% computational overhead.

Submission Number: 15