Cross-lingual transfer of multilingual models on low resource African Languages

ACL ARR 2025 February Submission4073 Authors

Abstract: Large transformer based multilingual models have significantly advanced natural language processing (NLP) research. However, their high resource demands and potential biases from diverse data sources have raised concerns about their effectiveness across low-resource languages. In contrast, monolingual neural models, trained on a single language, may better capture the nuances of the target language, potentially providing more accurate results. This study benchmarks the cross-lingual transfer capabilities from a high-resource language to a low-resource language for both, monolingual and multilingual models, focusing on Kinyarwanda and Kirundi, two Bantu languages. We evaluate the performance of transformer based architectures like Multilingual BERT (mBERT), AfriBERT, and BantuBERTa against traditional neural architectures such as BiGRU, CNN, and char-CNN. The models were trained on Kinyarwanda and tested on Kirundi datasets of news sentiment classification, with fine-tuning applied to assess the extent of performance improvement and catastrophic forgetting. AfriBERT achieved the highest cross-lingual accuracy of 88.3% after fine-tuning, while BiGRU emerged as the best-performing traditional model with 83.3% accuracy. We also analyze the degree of forgetting in the original language post-fine-tuning. While traditional monolingual models remain competitive, this study highlights that multilingual transformer models offer strong cross-lingual transfer capabilities by offering a comparative analysis between the both in resource limited settings.

Paper Type: Long

Research Area: Multilingualism and Cross-Lingual NLP

Research Area Keywords: cross-lingual transfer, multilingual representations, multilingual benchmarks, multilingual evaluation, less-resourced languages.

Contribution Types: Model analysis & interpretability, Approaches to low-resource settings, Data analysis

Languages Studied: Kinyarwanda, Kirundi

Submission Number: 4073