On the Impact of Matrix Language Identification for Automatic Speech Recognition of Code-Switched Speech

ACL ARR 2025 May Submission5716 Authors

Abstract: Code-switching (CS) is when a speaker alternates between two or more languages within a conversation, even within a single phrase. CS presents significant challenges for automatic speech recognition (ASR) systems due to mixed grammatical structures, accents and in-sentence language changes. One useful method for enhancing ASR performance on CS data is the accurate identification of the token Language Identities (LID). However, the LID of tokens do not explicitly inform ASR models of the dominant language which provides the grammatical structure for the CS utterance. The Matrix Language Frame (MLF) theory provides a syntactic and structural framework for the generation and analysis of CS utterances. It explains the CS process through the interaction of the two languages: the Matrix Language, which provides the grammatical structure for the CS utterance, and the Embedded Language, which is the language that is being inserted into the grammatical frame. This paper investigates the impact of Matrix Language Identity (MLID) analysis from the MLF theory on the effectiveness and accuracy of ASR systems when processing CS speech. The text-derived MLID was predicted from CS audio simultaneously with the ASR and token Language Identity (LID) prediction task, and the whole model was trained in a multi-task learning (MTL) setup. The proposed CS ASR system was compared to other MTL setups and showed a Mixed Error Rate (MER) decrease from 20.2\% in an Attention-CTC ASR baseline to 19.7\%. It was shown that having predicted MLID as Mandarin leads to an increase of recognised function words, indicating that MLID informs the ASR decoder of the grammatical properties of the utterance.

Paper Type: Long

Research Area: Multilingualism and Cross-Lingual NLP

Research Area Keywords: code-switching,software and tools

Contribution Types: NLP engineering experiment, Approaches to low-resource settings

Languages Studied: English,Mandarin

Submission Number: 5716