Enhancing Transformer Models for Igbo Language Processing: A Critical Comparative Study

Anthony Soronnadi, Olubayo Adekanmbi, Chinazo Anebelundu, David Ifeoluwa Adelani

Keywords: Transformer Models, Igbo NLP, IgboBERTa, NER, Topic Classification, Sentiment Analysis, Pretraining, Mask Language Modeling, Downstream Tasks

TL;DR: Enhancing Transformer Models for Igbo Language Processing

Abstract: This paper reports on an ongoing investigation aimed at reviewing and optimizing Transformer models for processing the African language Igbo, which has limited resources. Creating an effective language model is essential for enhancing NLP applications in this setting, given the specific challenges posed by Igbo's rich morphological structure, tonal system, and limited availability of digital resources. In order to investigate the adaptation and optimization of Transformer models and to improve the models for Igbo language processing, this work takes a critical comparison approach. First efforts have focused on developing a RoBERTa model pre-trained on clean Igbo text corpus, and evaluating its performance on downstream tasks such as named entity recognition, text classification, and sentiment analysis. In our evaluations across the above-mentioned NLP tasks, IgboBERTa demonstrates competitive or superior performance relative to larger models such as XLM-R-large, XLM-R-base, AfriBERTa, and AfroXLMR-base, particularly when considering its efficiency due to its smaller size of only 83.4M parameters. This efficiency makes IgboBERTa particularly appealing for resource-constrained environments common in African NLP applications.

Submission Number: 42