Variational Visual Question Answering for Uncertainty-Aware Selective Prediction

TMLR Paper6289 Authors

Abstract: Despite remarkable progress in recent years, vision language models (VLMs) remain prone to overconfidence and hallucinations on tasks such as Visual Question Answering (VQA) and Visual Reasoning. Bayesian methods can potentially improve reliability by helping models selectively predict, that is, models respond only when they are sufficiently confident. Unfortunately, Bayesian methods are often assumed to be costly and ineffective for large models, and there exists little evidence to show otherwise for multimodal applications. Here, we show the effectiveness and competitive edge of variational Bayes for selective prediction in VQA for the first time. We build on recent advances in variational methods for deep learning and propose an extension called "Variational VQA". This method improves calibration and yields significant gains for selective prediction on VQA and Visual Reasoning, particularly when the error tolerance is low (≤ 1%). Often, just one posterior sample can yield more reliable answers than those obtained by models trained with AdamW. In addition, we propose a new risk-averse selector that outperforms standard sample averaging by considering the variance of predictions. Overall, we present compelling evidence that variational learning is a viable option to make large VLMs safer and more trustworthy.

Submission Type: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

**Main paper changes (all are marked in blue in the pdf):**
- Added Table 3 for a comparison to Deep Ensembles
- Added a paragraph in Section 4.3 to explain the risk-averse Selector from a perspective of credible intervals
- A few clarifying sentences added throughout the paper in response to reviewers' requests

**Appendix changes (all are marked in blue in the pdf):**
- Reworked Appendix Section B about Training/Inference Overhead in terms of time and peak GPU memory
- Added Appendix Section F about Deep Ensembles
- Added Appendix Section G, which is about threshold generalization
- Added Appendix Section H, which is about comparing VarVQA to the Selector method from prior work

Assigned Action Editor: ~Andreas_Kirsch1

Submission Number: 6289