Isotropic Representation Can Improve Zero-Shot Cross-Lingual Transfer on Multilingual Language Models

Yixin Ji, Jikai Wang, Juntao Li, Hai Ye, Min Zhang

Submission Type: Regular Long Paper

Submission Track: Multilinguality and Linguistic Diversity

Submission Track 2: Language Modeling and Analysis of Language Models

Keywords: isotropic representation, cross-lingual, multilingual

Abstract:

With the development of multilingual pre-trained language models (mPLMs), zero-shot cross-lingual transfer shows great potential.
To further improve the performance of cross-lingual transfer, many studies have explored representation misalignment caused by morphological differences but neglected the misalignment caused by the anisotropic distribution of contextual representations.
In this work, we propose enhanced isotropy and constrained code-switching for zero-shot cross-lingual transfer to alleviate the problem of misalignment caused by the anisotropic representations and maintain syntactic structural knowledge.
Extensive experiments on three zero-shot cross-lingual transfer tasks demonstrate that our method gains significant improvements over strong mPLM backbones and further improves the state-of-the-art methods.\footnote{Our code will be available at \url{https://github.com/Dereck0602/IsoZCL}.}

Submission Number: 5438