Gender Bias in Vision-Language Assistants

Leander Girrbach, Yiran Huang, Isabel Rio-Torto, Stephan Alaniz, Trevor Darrell, Zeynep Akata

Keywords: social bias, vision-language models

TL;DR: We evaluate gender bias w.r.t. work-related soft skills in 16 popular vision-language assistants (VLAs).

Abstract: Pre-trained large language models (LLMs) have been reliably integrated with visual input for multimodal tasks. The widespread adoption of instruction-tuned vision-language assistants (VLAs) like LLaVA and MiniGPT, necessitates evaluating social biases. We measure gender bias in VLAs and evaluate 16 popular models regarding work-relevant skills. Specifically, given an image of either a man or a woman, we prompt the VLA whether the displayed person posses a given skill. Results show that many models exhibit bias towards associating work-relevant skills with females, although an image alone should not allow to make this assessment. Our research underscores the need for pre-deployment gender bias tests in VLAs and advocates for the development of debiasing strategies to ensure equitable societal outcomes.

Submission Number: 5