Do Large Language Models Treat South Asian Languages Equally? A Comparative Study of Bangla, Hindi, and Urdu in Low-Resource Settings

ACL ARR 2025 July Submission982 Authors

Abstract: Large language models (LLMs) have garnered significant interest in natural language processing (NLP), particularly their remarkable performance in various downstream tasks in resource-rich languages. Recent studies have highlighted the limitations of LLMs in low-resource languages, with a primary focus on binary classification tasks and minimal attention to South Asian languages. These limitations are primarily attributed to constraints such as dataset scarcity, computational costs, and specific research gaps for low-resource languages. To address this gap, we present new datasets for sentiment and hate speech tasks by translating from English to Bangla, Hindi, and Urdu, facilitating research in low-resource language processing. Further, we comprehensively examine zero-shot learning using multiple LLMs in English and widely spoken South Asian languages. Our findings indicate that GPT-4 consistently outperforms Llama 2 and Gemini, with English consistently demonstrating superior performance on diverse tasks compared to low-resource languages. Furthermore, our analysis reveals that natural language inference (NLI) exhibits the highest performance among the evaluated tasks, with GPT-4 demonstrating superior capabilities.

Paper Type: Short

Research Area: Multilingualism and Cross-Lingual NLP

Research Area Keywords: Multilingual NLP, Natural Language Inference, Sentiment Analysis, Hate speech, Model Evaluations

Contribution Types: Model analysis & interpretability, Data analysis

Languages Studied: Urdu, Bangla, Hindi

Previous URL: https://openreview.net/forum?id=0PnYWbffui

Reassignment Request Area Chair: No, I want the same area chair from our previous submission (subject to their availability).

Reassignment Request Reviewers: No, I want the same set of reviewers from our previous submission (subject to their availability)

Justification For Not Keeping Action Editor Or Reviewers: NA

A1 Limitations Section: This paper has a limitations section.

A2 Potential Risks: N/A

B Use Or Create Scientific Artifacts: Yes

B1 Cite Creators Of Artifacts: Yes

B1 Elaboration: Section 3

B2 Discuss The License For Artifacts: N/A

B3 Artifact Use Consistent With Intended Use: N/A

B4 Data Contains Personally Identifying Info Or Offensive Content: Yes

B4 Elaboration: Appendix C.1.1

B5 Documentation Of Artifacts: N/A

B6 Statistics For Data: Yes

B6 Elaboration: Appendix C.1.1

C Computational Experiments: Yes

C1 Model Size And Budget: No

C1 Elaboration: Due the page limitation we did not discuss the costs.

C2 Experimental Setup And Hyperparameters: Yes

C2 Elaboration: Section 3

C3 Descriptive Statistics: Yes

C3 Elaboration: Section 4 and Appendix

C4 Parameters For Packages: N/A

D Human Subjects Including Annotators: Yes

D1 Instructions Given To Participants: Yes

D1 Elaboration: Section B.1.1

D2 Recruitment And Payment: N/A

D3 Data Consent: N/A

D4 Ethics Review Board Approval: N/A

D5 Characteristics Of Annotators: Yes

D5 Elaboration: Appendix B.1.1

E Ai Assistants In Research Or Writing: No

E1 Information About Use Of Ai Assistants: N/A

Author Submission Checklist: yes

Submission Number: 982