Fairness-Aware Test-Time Prompt Tuning

ICLR 2026 Conference Submission18273 Authors

Keywords: test-time adaptation, test-time debiasing, prompt-tuning, vision-language models, algorithmic fairness

TL;DR: We introduce FairTPT, a novel fairness-aware test-time adaptation method for vision-language models that reduces bias by jointly optimizing target-attribute and sensitive-attribute entropy during episodic prompt tuning.

Abstract: Vision-language models have displayed remarkable capabilities in multi-modal understanding and are increasingly used in critical applications where economic and practical deployment constraints prohibit re-training or fine-tuning. However, these models can also exhibit systematic biases that disproportionately affect protected demographic groups and existing approaches to addressing these biases require extensive model retraining and access to demographic attributes. There is a clear need to develop test-time adaptation (TTA) approaches that improve the fairness characteristics of pretrained models under distributional shift. In this paper, we evaluate how episodic TTA affects fairness in CLIP classification under subpopulation shifts and develop FairTPT, a novel fairness-aware episodic TTA method that jointly minimizes target marginal entropy while maximizing spurious marginal entropy through soft-prompt tuning. We find that standard episodic TTA generally exacerbates disparities between majority and minority groups, that blinding a model to spurious attributes without degrading target performance is inherently challenging, and that excessive blinding can lead to catastrophic forgetting. This model collapse can be prevented by monitoring test-time changes in target loss within the linear regime, while still achieving fairness improvements on reactive data and preserving overall performance. Thus refined, FairTPT outperforms all state-of-the-art episodic test-time debiasing methods and establishes a foundation for robust TTA—essential for achieving fairness in practice.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 18273