CoLMbo: Speaker Language Model for Descriptive Profiling

Massa Baali, Shuo Han, Syed Abdul Hannan, Soham Deshmukh, Rita Singh, Bhiksha Raj

Keywords: speaker LM, language models, speaker verification, profiling, speaker identification

Presentation Preference: Open to it if recommended by organizers

Abstract:

Speaker recognition systems are often limited to classification tasks and struggle to generate detailed speaker characteristics or provide context-rich descriptions. These models primarily extract embeddings for speaker identification but fail to capture demographic
attributes such as dialect, gender, and age in a structured manner. This paper introduces CoLMbo, a Speaker Language Model (SLM) that addresses these limitations by integrating a speaker encoder with prompt-based conditioning. This allows for the creation of detailed captions based on speaker embeddings. CoLMbo utilizes user defined prompts to adapt dynamically to new speaker characteristics and provides customized descriptions, including regional dialect variations and age-related traits. This innovative approach not only enhances traditional speaker profiling but also excels in zero-shot scenarios across diverse datasets, marking a significant advancement in the field of speaker recognition. The code is available at: https://github.com/massabaali7/CoLMbo

Submission Number: 28