UNSUPERVISED CONFORMAL INFERENCE: BOOTSTRAPPING AND ALIGNMENT TO CONTROL LLM UNCERTAINTY

ICLR 2026 Conference Submission21698 Authors

Keywords: conformal prediction, uncertainty quantification, large language models, bootstrapping, Gram matrix, hallucination detection, calibration, alignment

TL;DR: We propose an unsupervised conformal framework for black-box LLMs: Gram-geometry scoring ,batched bootstrap calibration and conformal alignment, yielding near-nominal coverage and reliable factuality improvements.

Abstract: Deploying black-box LLMs requires managing uncertainty in the absence of token-level probability or true labels. We propose introducing an unsupervised conformal inference framework for generation, which integrates: generative models, incorporating: (i) an LLM-compatible atypical score derived from response-embedding Gram matrix, (ii) UCP combined with a bootstrapping variant (BB-UCP) that aggregates residuals to refine quantile precision while maintaining distribution-free, finite-sample coverage, and (iii) conformal alignment, which calibrates a single strictness parameter $\tau$ so a user predicate (e.g., factuality lift) holds on unseen batches with probability $\ge 1-\alpha$. Across different benchmark datasets, our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP, while consistently reducing the severity of factuality severity, outperforming lightweight per-response detectors with similar computational demands. The result is a label-free, API-compatible gate for test-time filtering that turns geometric signals into calibrated, goal-aligned decisions.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 21698