Improving Multilingual Capabilities with Cultural and Local Knowledge \\in Large Language Models While Enhancing Native Performance

ACL ARR 2025 July Submission1254 Authors

Abstract: Large Language Models (LLMs) have shown remarkable capabilities, but their development has primarily focused on English and other high-resource languages, leaving many languages underserved. We present our latest Hindi-English bilingual LLM with ~3\% average improvement in benchmark scores over both languages, outperforming models twice its size. Using a curated dataset composed of English and Hindi instruction data of 485K samples, we instruction-tuned models such as Qwen-2.5-14B-Instruct and Phi-4 to improve performance over both English and Hindi. Our experiments, encompassing seven different LLMs of varying parameter sizes and over 140 training attempts with varying English-Hindi training data ratios, demonstrated that it is possible to significantly improve multilingual performance without compromising native performance. Further, our approach avoids resource-intensive techniques like vocabulary expansion or architectural modifications, thus keeping the model size small. Our results indicate that modest fine-tuning with culturally and locally informed data can bridge performance gaps without incurring significant computational overhead. We release our training code, datasets, and models under MIT and Apache licenses to aid further research towards under-represented and low-resource languages.

Paper Type: Long

Research Area: Multilingualism and Cross-Lingual NLP

Research Area Keywords: New Language Adaptation, Multilingual LLMs, Indic LLM

Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Publicly available software and/or pre-trained models, Data resources

Languages Studied: English, Hindi

Submission Number: 1254