When Language Models Speak Burmese: Evaluating Hallucination in Low-Resource Domain Question Answering

Kushal Trivedi, Murtuza Shaikh, Sriyansh Sharma

Keywords: Hallucination, Low-Resource Languages, Domain-specific Question Answering

TL;DR: This research evaluates small and large language models on a low-resource, domain-specific dataset (Myanmar Mpox Q&A) to examine their tendency to hallucinate when answering factual questions.

Abstract: Language models have recently gained substantial attention in natural language processing, demonstrating strong performance across a wide range of tasks, including text classification, text generation, language modeling, and question answering (Q\&A). Despite these advancements, one of the most pressing challenges in language models is hallucination: the generation of responses that are fluent and plausible-sounding but factually incorrect, irrelevant, or fabricated. This study presents preliminary work investigating the impact of hallucination in Q\&A tasks for low-resource languages. Specifically, we evaluate model performance on the MPox-Myanmar dataset, employing both small- and large-scale language models accessible through APIs. Our research contributes by systematically examining observable hallucination across model sizes and prompting strategies, analyzing whether intuition about their behavior holds consistently, and providing explainability behind the observed patterns.

Submission Number: 17