KnowHalu: Multi-Form Knowledge Enhanced Hallucination Detection
Keywords: Hallucination Detection; Large Language Model; Factual Checking; Multi-Form Knowledge
TL;DR: We introduce "KnowHalu", a novel approach with two main phases (off-target hallucination detection and multi-step factual checking) for detecting hallucinations in text generated by LLMs, leveraging multi-form knowledge for factual checking.
Abstract: As large language models (LLMs) become increasingly integral to a wide array of applications, ensuring the factual accuracy of their outputs and mitigating hallucinations is paramount. Current approaches, which primarily rely on self-consistency checks or post-hoc fact-checking, often fall short by disregarding the nuanced structure of queries and the diverse forms of contextual knowledge required for accurate response generation.
To address these shortcomings, we introduce KnowHalu (pronounced “No Halu”), the first multi-form knowledge-based hallucination detection framework. We also introduce a new category of hallucinations, off-target hallucinations, which occur when responses are factually accurate but irrelevant or nonspecific to the query (e.g., answering "What’s the primary language in Barcelona?" with "European language").
In particular, KnowHalu employs a rigorous two-phase process to detect hallucinations. In the first phase, it isolates off-target hallucinations by analyzing the semantic alignment between the response and the query. In the second phase, it conducts a novel multi-form knowledge-based fact-checking through a comprehensive pipeline of reasoning and query decomposition, knowledge retrieval, knowledge form optimization, judgment generation, and judgment aggregation.
Extensive evaluations demonstrate that KnowHalu significantly surpasses state-of-the-art (SOTA) baselines across diverse tasks, achieving over 15% improvement in question answering (QA) and 6% in summarization tasks when applied to the same underlying LLM. These results underscore the effectiveness and versatility of KnowHalu, setting a new benchmark for hallucination detection and paving the way for safer and more reliable LLM applications.
Primary Area: foundation or frontier models, including LLMs
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 10095
Loading