IUQ: Interrogative Uncertainty Quantification for Long-form Generation

ACL ARR 2025 May Submission4650 Authors

Abstract: Large Language Models (LLMs) have seen remarkable development but are still prone to hallucination. Developing robust and comprehensive Uncertainty Quantification (UQ) approaches for long-form text generation remains a major challenge. In this paper, we present Interrogative Uncertainty Quantification (IUQ), a novel self-consistency based UQ approach that leverages the language model's tendency to generate semantically coherent yet factually incorrect responses. IUQ builds its estimation on both the trustworthiness of individual facts and their contextual consistency within the model generation. By prompting the language model to go through an interrogate-respond process, IUQ can reliably measure generation-level uncertainties in addition to the model's overall tendency to hallucinate. We evaluate our method with the latest models over diverse model families, and observe a consistent gain in classification metrics.

Paper Type: Long

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: calibration/uncertainty,counterfactual/contrastive explanations,free-text/natural language explanations,knowledge tracing/discovering/inducing

Contribution Types: Model analysis & interpretability, Data analysis

Languages Studied: English

Submission Number: 4650