Self-Supervised Variational Priors for Robust Bayesian Inference

Erik Englesson, Iaroslav Melekhov, Juho Kannala, Hossein Azizpour

Keywords: Bayesian Deep Learning; Variational Inference; Self-Supervised Learning

TL;DR: We propose to replace the standard priors with self-supervised variational posteriors. Specifically, we propose a variational method based on instance discrimination.

Abstract: The standard in Bayesian Deep Learning is to use zero-mean isotropic Gaussian priors over neural network weights. When the likelihood is reliable (\eg, with abundant clean labels), these priors matter little. However, when the likelihood is unreliable due to imperfect data, the weakness of standard priors becomes apparent, resulting in poor robustness and uncertainty estimates. To address this, we propose to replace the standard priors with self-supervised variational posteriors. Specifically, we propose a variational method based on instance discrimination. Our objective is derived from the evidence lower bound and made scalable through approximations. Empirically, our priors enhance data-efficiency, robustness to label noise, generalization and calibration on the CIFAR datasets.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 258