hULMonA: The Universal Language Model in Arabic

Obeida ElJundi, Wissam Antoun, Nour El Droubi, Hazem Hajj, Wassim El-Hajj, Khaled Shaban

Abstract:

Arabic is a complex language with limited resources which makes it challenging to produce accurate text classification tasks such as
sentiment analysis. The utilization of transfer learning (TL) has recently shown promising results for advancing the accuracy of text classification in English. TL models are pre-trained on large corpora and then fine-tuned on task-specific datasets. In particular, universal language models (ULMs), such as recently developed BERT, have achieved state-of-the-art results in various NLP tasks in English. In this
paper, we hypothesize that similar success can be achieved for Arabic. The work aims at supporting the hypothesis by developing the first
Universal Language Model in Arabic (hULMonA - meaning our dream), demonstrating its use for Arabic classifications tasks, and demonstrating how a pre-trained multilingual BERT can also be used for Arabic. We then conduct a benchmark study to evaluate
both ULM successes with Arabic sentiment analysis. Experiment results show that the developed hULMonA and multi-lingual ULM are able to generalize well to multiple Arabic data sets and achieve new state-of-the-art results in Arabic Sentiment Analysis for some of the tested sets.