Large Language Models are Biased to Detect Hallucination across Languages
Abstract: Hallucinations in generative AI, particularly in large language models (LLMs), have emerged as a significant concern. These models often facilitate multilingual operations, including querying and conversation. Yet, few research efforts have been devoted to understanding hallucinations in a multilingual context, specifically regarding the equitable treatment of supported languages, due to the lack of available benchmarks. Addressing this gap, this paper first proposes Poly-FEVER, a large-scale publicly accessible multilingual fact extraction and verification dataset for hallucination detection that covers 11 languages and more than 800K fact claims with diverse topics. We utilize Poly-FEVER to evaluate the hallucination detection capabilities of ChatGPT and LLaMA-2 series.
Our investigation extends to exploring hallucination causes, employing Latent Dirichlet Allocation (LDA) for topic distribution analysis and web searches to assess resource imbalances.
Furthermore, we propose a mitigation approach combining linguistic adjustments and resource-oriented strategies, including a trained LDA model and the Retrieval Augmented Generation (RAG) approach, to enhance the robustness and reliability of multilingual information verification in LLMs.
Our findings highlight the critical need for multilingual benchmarks Poly-FEVER and demonstrate the potential of mitigation strategy to address biased detection abilities on hallucinations, thus contributing to the development of more equitable and reliable multilingual LLMs.
Paper Type: Long
Research Area: Semantics: Lexical and Sentence-Level
Research Area Keywords: AI Hallucination, Multilingual bias, Large language models (LLMs), Poly-FEVER dataset
Contribution Types: NLP engineering experiment, Data resources
Languages Studied: English, Mandarin Chinese, Hindi, Arabic, Bengali, Japanese, Korean, Tamil, Thai, Georgian, Amharic
Submission Number: 3892
Loading