CA-BED: Conversation-Aware Bayesian Experimental Design

Daniel Arnould, Rashad Aziz, Zixuan Kang, Tanav Changal, Kevin Zhu, Sunishchal Dev, Gabriel Grand, Shreyas Sunil Kulkarni

Keywords: Uncertainty, Bayesian, BED, Information-seeking, Information-acquiring, QA, Question-Answering, Information-Theory, Tree, Planning, Reliability, Reasoning

Abstract: Large Language Models (LLMs) excel at static reasoning tasks, yet their reliability diminishes in interactive scenarios where information must be actively acquired through questioning. A key challenge lies in asking questions that minimize uncertainty while simultaneously internalizing responses that may be ambiguous or partially specified. To address this, we propose Conversation-Aware Bayesian Experimental Design (CA-BED), an inference-time probabilistic dialogue planning framework that integrates Bayesian Experimental Design with LLM-based likelihood estimation to optimize question selection over multiple conversational turns. CA-BED models uncertainty continuously, anticipates potential answers, and propagates expected information gain through a simulated conversation tree, enabling more efficient and robust information acquisition. Across entity-deduction benchmarks, CA-BED yields an average 25% improvement in success rates over direct prompting, with comparable gains relative to alternative information-seeking methods. It further achieves a 10.25% average reduction in conversation length, indicating greater efficiency in information gathering. These results highlight CA-BED’s effectiveness as a principled framework for reliable interactive reasoning in real-world settings.

Submission Number: 50