When Alignment Hurts: Decoupling Representational Spaces in Multilingual Models

Ahmed Elshabrawy, Hour Kaing, Haiyue Song, Alham Fikri Aji, Hideki Tanaka, Masao Utiyama, Raj Dabre

Keywords: Causal Representation Learning, Multilingual Language Models, Representation Disentanglement, Low-resource Language Modeling, Subspace Probing, Generative Modeling

TL;DR: Reducing representational entanglement with high-resource languages improves generative modeling for related low-resource varieties through causal subspace interventions.

Abstract: It is often assumed that aligning low-resource varieties with high-resource standards improves multilingual modeling in large language models (LLMs). We challenge this view with the first intervention-based study showing that excessive representational entanglement with dominant varieties can degrade generative quality in machine translation, suggesting a causal link between representational dominance and weaker downstream performance on low-resource varieties. We introduce an online variational probing fine-tuning method that continuously estimates the subspace of a dominant variety during generative fine-tuning (mainly translation) and penalizes it to reduce its span. Across six language families, reducing alignment consistently improves low-resource translation quality, with gains of up to +11.7 ChrF++ / +10.1 COMET for European Portuguese, +5.3 / +4.3 for Indonesian, +4.6 / +4.2 for Kven Finnish, and +2.7 / +2.1 for Low German. In Arabic, several dialects improve by up to +4.7 ChrF++ and +1.4 COMET despite sharp drops for cross-lingual tasks (e.g., translation to MSA, English, or French), suggesting that the effect extends beyond simple cross-lingual alignment. Alongside these intervention results, we present qualitative and geometric analyses that further support our hypothesis. Together, our findings show that disentangling high-resource subspaces can unlock representational capacity for related low-resource varieties and provide a practical means of controlling representational allocation in multilingual LLMs.

Primary Area: interpretability and explainable AI

Submission Number: 17848