Uncertainty-Aware Search and Value Models in LLMs

Fei Yu, Yingru Li, Benyou Wang

Keywords: uncertainty, search, verifier

TL;DR: we introduce uncertainty qualification to mitigate the verifier failures

Abstract: Value model-guided search is effective in steering the generation but suffers from verifier failures: imperfect verifiers may mistakenly prune all the valid paths. This limitation could arise from reliability degradation in value models in unseen reasoning paths. To address this, we propose an uncertainty-aware search framework that includes two key components: (1) uncertainty-aware value models that incorporate uncertainty into predictions, and (2) an uncertainty-aware selection process using the proposed efficient Group Thompson Sampling algorithm. Experiments on two In-Distribution (ID) settings (GSM8K and MATH) and three Out-Of-Distribution (OOD) settings (e.g. AIME25 and Minerva Math) show that our method mitigates verifier failures. This work establishes the first systematic integration of uncertainty quantification in LLM search paradigms.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 24166