If there is no underfitting, there is no Cold Posterior Effect

Yijie Zhang, Yi-Shan Wu, Luis A. Ortega, Andres R Masegosa

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Bayesian neural networks, model misspecification, prior misspecification, cold posterior effect

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We show that misspecification leads to Cold Posterior Effect (CPE) only when the resulting Bayesian posterior underfits. We theoretically show that if there is no underfitting, there is no CPE.

Abstract: The cold posterior effect (CPE) (Wenzel et al., 2020) in Bayesian deep learning shows that, for posteriors with a temperature T<1, the resulting posterior predictive could have better performances than the Bayesian posterior (T=1). In recent years, there have been several main hypotheses to explain CPE: prior misspecification, likelihood misspecification and data augmentation. In this work, we show a more nuanced understanding of the CPE as we show that \emph{misspecification leads to CPE only when the resulting Bayesian posterior underfits}. In fact, we theoretically show that if there is no underfitting, there is no CPE.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6387