On Evaluation of Bangla Word Analogies

Mousumi Akter, Souvika Sarkar, Shubhra Kanti Karmaker Santu

Submission Type: Regular Short Paper

Submission Track: Resources and Evaluation

Keywords: Bangla, Word Analogy, Evaluation

TL;DR: This paper presents a benchmark dataset of Bangla word analogies for evaluating the quality of existing Bangla word embeddings.

Abstract: This paper presents a benchmark dataset of Bangla word analogies for evaluating the quality of existing Bangla word embeddings. Despite being the 7th largest spoken language in the world, Bangla is still a low-resource language and popular NLP models often struggle to perform well on Bangla data sets. Therefore, developing a robust evaluation set is crucial for benchmarking and guiding future research on improving Bangla word embeddings, which is currently missing. To address this issue, we introduce a new evaluation set of 16,678 unique word analogies in Bangla as well as a translated and curated version of the original Mikolov dataset (10,594 samples) in Bangla. Our experiments with different state-of-the-art embedding models reveal that current Bangla word embeddings struggle to achieve high accuracy on both data sets, demonstrating a significant gap in multilingual NLP research.

Submission Number: 3266