Scalable Intersectional Bias Auditing in Vision-Language Models through Combinatorial Interaction Testing

Heejin Bin, Junyoung Choi, JangHyun Kim, Seungjae Kim, Shin Yoo

Track: Main Papers Track (6 to 9 pages)

Keywords: Intersectional Fairness, Combinatorial Interaction Testing, Vision-Language Models, Bias Auditing, Synthetic Data Generation, Algorithmic Fairness, Scalable Auditing, Diffusion Models

TL;DR: We propose a scalable auditing framework for VLMs that leverages Combinatorial Interaction Testing and diffusion-based synthetic data to systematically detect and analyze higher-order intersectional biases across diverse identity combinations.

Abstract: Intersectionality analysis is critical for algorithmic fairness, since individuals hold multiple and overlapping identities, leading to unique challenges and biases. However, scalable detection of intersectional fairness bugs, i.e., systematic misbehaviors that emerge only in higher-order intersectional subgroups, remains difficult due to the scarcity of data across diverse identity combinations. We propose a scalable auditing framework for intersectional biases in Vision-Language Models (VLMs): our framework is based on Combinatorial Interaction Testing (CIT) and diffusion models. CIT enables systematic sampling of all $t$-way identity interactions with a minimal set of test suites, while diffusion allows us to generate specific inputs for VLMs that fit the given combination of identities. By integrating CIT with synthetic image generation, we substantially mitigate the computational and generative burden, making the exploration of deeply nested subgroups tractable and scalable. Our empirical evaluation shows that our approach can flexibly balance subgroup specificity with test efficiency, uncovering compounding biases that remain invisible to conventional univariate or bivariate assessments.

Submission Number: 53