Uncertainty Quantification and Decomposition for LLM-based Recommendation

WONBIN KWEON, Sanghwan Jang, SeongKu Kang, Hwanjo Yu

Track: User modeling, personalization and recommendation

Keywords: Recommendation, Large Language Models, Uncertainty

TL;DR: We propose a novel framework for measuring the uncertainty of LLM-based recommendations and provide some findings.

Abstract:

Despite the widespread adoption of large language models (LLMs) for recommendation, we demonstrate that LLMs often exhibit uncertainty in their recommendations.
To ensure the trustworthy use of LLMs in generating recommendations, we emphasize the importance of assessing the reliability of recommendations generated by LLMs.
We start by introducing a novel framework for estimating the predictive uncertainty to quantitatively measure the reliability of LLM-based recommendations.
We further propose to decompose the predictive uncertainty into recommendation uncertainty and prompt uncertainty, enabling in-depth analyses of the primary source of uncertainty.
Through extensive experiments, we (1) demonstrate predictive uncertainty effectively indicates the reliability of LLM-based recommendations, (2) investigate the origins of uncertainty with decomposed uncertainty measures, and (3) propose uncertainty-aware prompting for a lower predictive uncertainty and enhanced recommendation.
Our source code and model weights are available at https://anonymous.4open.science/r/UNC_LLM_REC

Submission Number: 1375