Mixture Distributions for Scalable Bayesian Inference

Pranav Poduval, Hrushikesh Loya, Rajat Patel, Sumit Jain

TL;DR: Scalable Bayesian NN's Alternative to Deep Ensemble

Abstract:

Bayesian Neural Networks (BNNs) provides a mathematically grounded framework to quantify uncertainty. However BNNs are computationally inefficient,
thus are generally not employed on complicated machine learning tasks. Deep
Ensembles were introduced as a Bootstrap inspired frequentist approach to the
community, as an alternative to BNN’s. Ensembles of deterministic and stochastic networks are a good uncertainty estimator in various applications (Although,
they are criticized for not being Bayesian). We show Ensembles of deterministic
and stochastic Neural Networks can indeed be cast as an approximate Bayesian
inference. Deep Ensembles have another weakness of having high space complexity, we provide an alternative to it by modifying the original Bayes by Backprop (BBB) algorithm to learn more general concrete mixture distributions over
weights. We show our methods and its variants can give better uncertainty estimates at a significantly lower parametric overhead than Deep Ensembles. We
validate our hypothesis through experiments like non-linear regression, predictive
uncertainty estimation, detecting adversarial images and exploration-exploitation
trade-off in reinforcement learning.

Keywords: uncertainty estimation, Deep Ensembles, Adverserial Robustness

Code: https://drive.google.com/open?id=1a-tweSF-WdpRUOJPPgnfK0lu4CnYHoKy