VIB is Half Bayes

Alexander A Alemi, Warren Richard Morningstar, Ben Poole, Ian Fischer, Joshua V. Dillon

Keywords: Variational Information Bottleneck, Information Bottleneck, Bayesian Inference, PAC-Bayes, Statistical Learning Theory

TL;DR: The Variational Information Bottleneck can rederived as Half-Bayesian.

Abstract: In discriminative settings such as regression and classification there are two random variables at play, the inputs $X$ and the targets $Y$. Here, we demonstrate that the Variational Information Bottleneck can be viewed as a compromise between fully empirical and fully Bayesian objectives, attempting to minimize the risks due to finite sample of $Y$ only. We argue that this approach provides the some of the benefits of Bayes while requiring only some of the work

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2011.08711/code)