Seeds, Contexts, and Tongues: Decoding the Drivers of Hallucination in Language Models

ICLR 2025 Workshop BuildingTrust Submission143 Authors

Track: Long Paper Track (up to 9 pages)

Keywords: Language Models (LLMs), Hallucination Detection, Semantic Entropy, Natural Language Processing (NLP), Pidgin Language, Cross-Lingual Analysis

TL;DR: We demonstrate that in free-form text production, hallucinations in LLMs occur at different ends of the hyperparameter scale, with the ideal range dependent on prompt language and cultural context.

Abstract: This study investigates hallucinations in Large Language Models (LLMs) during free-form text generation, particularly in Nigerian and Western contexts. We study how hyperparameters, cultural background, and prompt language (particularly, Nigerian Pidgin) affect hallucination rates. Using semantic entropy as an indicator of hallucination, we examine response variability in Llama 3.1 outputs and cluster them using the entailment model microsoft/deberta-base-mnli to identify semantic similarity. We then use these clusters to calculate semantic entropy (the variation in meanings of the LLM's responses) using a variant of Shannon entropy to quantify hallucination likelihood. Our findings shed light on ways to improve LLM reliability and consistency across linguistic and cultural situations.

Submission Number: 143