Cross-Lingual Transfer with Class-Weighted Language-Invariant Representations

Ruicheng Xian, Heng Ji, Han Zhao

Keywords: cross-lingual transfer, unsupervised cross-lingual learning, multilingual neural language model, domain adaptation

Abstract: Recent advances in neural modeling have produced deep multilingual language models capable of extracting cross-lingual knowledge from non-parallel texts and enabling zero-shot downstream transfer. While their success is often attributed to shared representations, quantitative analyses are limited. Towards a better understanding, through empirical analyses, we show that the invariance of feature representations across languages—an effect of shared representations—strongly correlates with transfer performance. We also observe that distributional shifts in class priors between source and target language task data negatively affect performance, a largely overlooked issue that could cause negative transfer with existing unsupervised approaches. Based on these findings, we propose and evaluate a method for unsupervised transfer, called importance-weighted domain alignment (IWDA), that performs representation alignment with prior shift estimation and correction using unlabeled target language task data. Experiments demonstrate its superiority under large prior shifts, and show further performance gains when combined with existing semi-supervised learning techniques.

One-sentence Summary: We propose an importance-weighted domain adaptation method for unsupervised cross-lingual learning that is effective at correcting class prior shifts, a distributional property that negatively affects transfer performance.