Self-Supervised Variational Priors for Robust Bayesian Inference

Erik Englesson, Iaroslav Melekhov, Juho Kannala, Hossein Azizpour

Abstract: The standard in Bayesian Deep Learning is to use zero-mean isotropic Gaussian priors over neural network weights. When the likelihood is reliable (e.g., with abundant clean labels), these priors matter little. However, when the likelihood is unreliable due to imperfect data, the weakness of standard priors becomes apparent, resulting in poor robustness and uncertainty estimates. To address this, we propose to replace the standard priors with self-supervised variational posteriors. Specifically, we propose a variational method based on instance discrimination. Our objective is derived from the evidence lower bound and made scalable through approximations. Empirically, our priors enhance data-efficiency, robustness to label noise, generalization and calibration on the CIFAR datasets.

Keywords: Bayesian Deep Learning; Variational Inference; Self-Supervised Learning

TLDR: We propose to replace the standard priors with self-supervised variational posteriors. Specifically, we propose a variational method based on instance discrimination.

Submission Number: 28