VIB is Half BayesDownload PDF

Nov 23, 2020 (edited Jan 11, 2021)AABI2020Readers: Everyone
  • Keywords: Variational Information Bottleneck, Information Bottleneck, Bayesian Inference, PAC-Bayes, Statistical Learning Theory
  • TL;DR: The Variational Information Bottleneck can rederived as Half-Bayesian.
  • Abstract: In discriminative settings such as regression and classification there are two random variables at play, the inputs $X$ and the targets $Y$. Here, we demonstrate that the Variational Information Bottleneck can be viewed as a compromise between fully empirical and fully Bayesian objectives, attempting to minimize the risks due to finite sample of $Y$ only. We argue that this approach provides the some of the benefits of Bayes while requiring only some of the work
1 Reply