What do Geometric Hallucination Detection Metrics Actually Measure?

Eric Yeats, John Buckheit, Sarah McGuire Scullen, Brendan Kennedy, Loc Truong, Davis Brown, Bill Kay, Cliff Joslyn, Tegan Emerson, Michael J. Henry, John Anthony Emanuello, Henry Kvinge

Keywords: Hallucination detection, the geometry of hidden activations, large language models

TL;DR: We analyze what geometric hallucination detectors are actually detecting, our results suggest a novel normalization scheme that allows detectors to better transfer between subject domains.

Abstract: Hallucination remains a barrier to deploying generative models in high-consequence applications. This is especially true in cases where external ground truth is not readily available to validate model outputs. This situation has motivated the study of geometric signals in the internal state of an LLM that are predictive of hallucination and require limited external knowledge. Given that there are a range of factors that can lead model output to be called a hallucination (e.g., irrelevance vs incoherence), in this paper we ask what specific properties of a hallucination these geometric statistics actually capture. To assess this, we generate a synthetic dataset which varies distinct properties of output associated with hallucination. This includes output *correctness*, *confidence*, *relevance*, *coherence*, and *completeness*. We find that different geometric statistics capture different types of hallucinations. Along the way we show that many existing geometric detection methods have substantial sensitivity to shifts in task domain (e.g., math questions vs. history questions). Motivated by this, we introduce a simple normalization method to mitigate the effect of domain shift on geometric statistics, leading to AUROC gains of +34 points in multi-domain settings.

Submission Number: 54