From Bandit Regret to FDR Control: Online Selective Generation with Feedback Unlocking

Minjae Lee, Yoonjae Jung, Sangdon Park

Keywords: online learning, selective prediction, adversarial bandit, partial feedback, LLMs, hallucination

TL;DR: We propose ExSUL, an online learning framework for selective generation with adversarial bandit feedback that ensures FDR control. for reliable LLMs by unlocking learning signals from partial feedback.

Abstract:

As interactive generative systems are increasingly deployed in real-world applications, their tendency to generate unreliable or false responses raises serious concerns.
Selective generation mitigates this risk by ensuring that the system answers only when confident.
However, real-world deployments typically provide only partial user feedback on the selected response (e.g., thumbs up/down) and often operate in non-stationary or adversarial environments, for which effective learning methods are largely missing.
To bridge this gap, we propose ExSUL, a novel online learning framework for selective generation with adversarial bandit feedback.
Technically, we introduce
(i) a novel conversion lemma that translates the regret of any bandit algorithm into an FDR bound, and
(ii) feedback unlocking, a strategy that exploits the structure of selective generation to extract additional learning signals from partial feedback.
We prove that ExSUL achieves a regret bound of $\mathcal{O}(\sqrt{T \ln |\mathcal{H}|})$, matching the efficiency and FDR controllability of full-information settings despite receiving only partial feedback.
While applicable to general generative tasks, we demonstrate the efficacy of ExSUL for ensuring the reliability of Large Language Models (LLMs) through empirical validation on question-answering tasks across diverse non-stationary and adversarial settings.
Our results demonstrate that ExSUL robustly controls the FDR while maintaining competitive answering coverage.

Submission Number: 26