GradSim: Gradient-Based Language Grouping for Effective Multilingual Training

Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, Hinrich Schuetze

Submission Type: Regular Long Paper

Submission Track: Machine Learning for NLP

Submission Track 2: Multilinguality and Linguistic Diversity

Keywords: Effective multilingual learning, language grouping, gradient-based similarity

TL;DR: The paper proposes a gradient-based language grouping approach for multilingual training, and shows its superior performance compared to other language similarity measures in three diverse multilingual datasets.

Abstract: Most languages of the world pose low-resource challenges to natural language processing models. With multilingual training, knowledge can be shared among languages. However, not all languages positively influence each other and it is an open research question how to select the most suitable set of languages for multilingual training and avoid negative interference among languages whose characteristics or data distributions are not compatible. In this paper, we propose GradSim, a language grouping method based on gradient similarity. Our experiments on three diverse multilingual benchmark datasets show that it leads to the largest performance gains compared to other similarity measures and it is better correlated with cross-lingual model performance. As a result, we set the new state of the art on AfriSenti, a benchmark dataset for sentiment analysis on low-resource African languages. In our extensive analysis, we further reveal that besides linguistic features, the topics of the datasets play an important role for language grouping and that lower layers of transformer models encode language-specific features while higher layers capture task-specific information.

Submission Number: 3241