Sparsifying Bayesian neural networks with latent binary variables and normalizing flows
Abstract: Artificial neural networks are powerful machine learning methods used in many modern ap- plications. A common issue is that they have millions or billions of parameters, and therefore tend to overfit. Bayesian neural networks (BNN) can improve on this since they incorpo- rate parameter uncertainty. Latent binary Bayesian neural networks (LBBNN) further take into account structural uncertainty by allowing the weights to be turned on or off, enabling inference in the joint space of weights and structures. In this paper, we will consider two extensions of variational inference for the LBBNN: Firstly, by using the local reparametriza- tion trick (LRT), we improve on computational efficiency. Secondly, and more important, by using normalizing flows on the variational posterior distribution of the LBBNN parameters, we learn a more flexible variational posterior than the mean field Gaussian. Experimental results on real data show that this improves on predictive power compared to using mean field variational inference on the the LBBNN method, while also obtaining sparser networks. We also perform two simulation studies. In the first, we consider variable selection in a lo- gistic regression setting, where the more flexible variational distribution improves results. In the second study, we compare predictive uncertainty based on data generated from two- dimensional Gaussian distributions. Here, we argue that our Bayesian methods lead to more realistic estimates of predictive uncertainty.
Submission Length: Long submission (more than 12 pages of main content)
Previous TMLR Submission Url: https://openreview.net/forum?id=58jTKZb86S&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DTMLR%2FAuthors%23your-submissions)
Changes Since Last Submission: Revision of the manuscript and the supplementary according to the comments from the reviewers
Assigned Action Editor: ~Pierre_Alquier1
Submission Number: 2153
Loading