ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapters

Anonymous

Abstract: Our goal is to achieve high zero shot performance on low-resource languages (LRLs) that are unseen by the multilingual pre-trained language models like mBERT and XLM-R. A recent approach for handling LRLs is to use language adapters (LAs), but they are also unavailable for unseen languages. All existing works that study LAs for unseen languages train on only a single source language (English), and most use only the English adapter at test time. We believe that to achieve best zero-shot performance, we must make use of multiple (related) source languages/adapters at both training and test time. In response, we propose an architecture that performs both training-time and test-time ensembling of LAs. It also incorporates the typological properties of languages (encoded in existing language vectors) for further improvements. Experiments on four language families demonstrate substantial improvements over standard fine tuning and other recent baselines on sequence labelling tasks.

Paper Type: long

Research Area: Multilinguality