BengaliLLama: An Instruction Following LLaMA Model for Bengali

Anonymous

Abstract: In the field of Large Language Models (LLMs), significant advancements have predominantly focused on a limited set of languages, raising concerns in linguistically diverse regions such as India, where a wide array of regional languages are spoken, and the majority of individuals communicate in native languages other than English. Addressing this limitation, our study introduces BengaliLlama, a model tailored for Bengali, the world's seventh most widely spoken language. This research leverages a dataset of 252K Bengali instructions, translated and manually validated from various open-source resources, and employs the LoRA architecture and LLaMA for fine-tuning. The resulting BengaliLlama model demonstrates enhanced proficiency in processing and responding to instruction-based queries in Bengali. The study discussed comprehensive evaluations that will motivate various Indic Model studies in the future. BengaliLlama will be made available for research and non-commercial use, contributing to the broader goal of creating more linguistically diverse and accessible AI technologies.

Paper Type: short

Research Area: Multilinguality and Language Diversity

Contribution Types: Model analysis & interpretability, Reproduction study, Approaches to low-resource settings, Data resources, Surveys

Languages Studied: Bengali