Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models

ACL ARR 2025 February Submission2170 Authors

Abstract:

Multilingual language models (MLMs) store factual knowledge across languages but often struggle with cross-lingual factual consistency, i.e., with providing consistent responses to semantically equivalent prompts in different languages. While previous studies point out this issue, the underlying causes remain unexplored. In this work, we use mechanistic interpretability methods to investigate cross-lingual inconsistencies in MLMs. We find that MLMs encode knowledge in an language-independent concept space through most layers,
and only transition to language-specific spaces in the final layers. Failures during this language transition process often result in incorrect predictions in the target language, even when the model correctly predicts the answer in other languages.
To mitigate this inconsistency issue, we propose a linear shortcut method that bypasses computations in the final layers, enhancing both prediction accuracy and cross-lingual consistency. Overall, this study deepens the understanding of MLM mechanisms and offers insights for generating consistent factual predictions.

Paper Type: Long

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: cross-lingual transfer, knowledge tracing/discovering/inducing, probing, cross-lingual transfer, multilingualism

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: Arabic, Catalan, Greek, English, Spanish, Persian, French, Hebrew, Hungarian, Japanese, Korean, Dutch, Russian, Turkish, Ukrainian, Vietnamese, Chinese

Submission Number: 2170