- Keywords: Variational Information Bottleneck, Information Bottleneck, Bayesian Inference, PAC-Bayes, Statistical Learning Theory
- TL;DR: The Variational Information Bottleneck can rederived as Half-Bayesian.
- Abstract: In discriminative settings such as regression and classification there are two random variables at play, the inputs $X$ and the targets $Y$. Here, we demonstrate that the Variational Information Bottleneck can be viewed as a compromise between fully empirical and fully Bayesian objectives, attempting to minimize the risks due to finite sample of $Y$ only. We argue that this approach provides the some of the benefits of Bayes while requiring only some of the work