The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer

Anonymous

Abstract: We evaluate effectiveness of an existing approach to cross-lingual adjustment of mBERT using four typologically different languages (Spanish, Russian, Vietnamese, and Hindi) and three NLP tasks (QA, NLI, and NER). The adjustment uses a small parallel corpus to make embeddings of related words across languages similar to each other. It improves NLI in four languages and NER in three languages, while QA performance never improves and sometimes degrades. Analysis of distances between contextualized embeddings of related and unrelated words across languages showed that fine-tuning leads to ``foregetting'' some of the cross-lingual alignment information, which---we conjecture---can negatively affect the effectiveness of the zero-shot transfer. Based on this observation, we further improved performance on NLI using continual learning. Our study contributes to a better understanding of cross-lingual transfer capabilities of large multilingual language models and of effectiveness of their cross-lingual adjustment in various NLP tasks.

Paper Type: long

Editor Reassignment: yes

Reviewer Reassignment: yes