Acoustic-based Gender Differentiation in Speech-aware Language Models

Junhyuk Choi, Jihwan Seol, Nayeon Kim, chanhee cho, EunBin Cho, Bugeun Kim

Keywords: Gender Bias, Spoken Language Model, Acoustic-based Gender Differentiation, Fairness

Abstract: Speech-aware Language Models (SpeechLMs) have fundamentally transformed human-AI interaction by enabling voice-based communication, yet they may exhibit acoustic-based gender differentiation where identical questions lead to different responses based on the speaker's gender. However, this differentiation is not inherently binary; it demands a nuanced understanding of when acoustic cues serve as valid context versus when they result in algorithmic unfairness. To address this challenge, it is essential to distinguish between inappropriate bias and necessary personalization. To enable such an ethically balanced evaluation, we propose a new dataset of 9,208 speech samples constructed across three distinct contexts: Gender-Independent, Gender-Stereotypical, and Gender-Dependent. Our evaluation of the LLaMA-Omni series reveals a paradoxical pattern; models consistently exhibit male-oriented bias in Gender-Stereotypical questions despite requiring neutrality, while they failed to provide appropriate gender-differentiated responses in Gender-Dependent questions where differentiation is considerable. We confirm that this pattern persists regardless of neutral response options or voice neutralization techniques. Through a comparative analysis with backbone LLMs and an investigation of internal representations, we suspect that these biases primarily stem from the Whisper speech encoder whose encoding discerns different semantic content more clearly than gender characteristics. Our findings suggest that current SpeechLMs prioritize general fairness principles over contextual appropriateness, highlighting the critical need to move beyond monolithic bias removal toward context-aware acoustic alignment in future speechLMs.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 2807