Towards Efficient Posterior Sampling in Deep Neural Networks via Symmetry Removal

Jonas Gregor Wiese, Lisa Wimmer, Bernd Bischl, Stephan Günnemann, David Rügamer

Abstract: Bayesian inference in deep neural networks is challenging due to the high-dimensional, strongly multi-modal posterior landscape. Markov Chain Monte Carlo approaches asymptotically recover the true, intractable posterior but are prohibitively expensive for large modern architectures. Local posterior approximations, while often yielding satisfactory results in practice, crudely disregard the posterior geometry. We propose to exploit well-known parameter symmetries induced by neuron interchangeability and output activation to retrieve a drastically reduced -- yet exact -- posterior over uniquely identified parametrizations. To this end, we provide an algorithm for explicit symmetry removal and develop an upper bound on Monte Carlo samples required to capture the reduced posterior. Our experiments suggest that efficient sampling from the functionally relevant part of the posterior is indeed possible, opening up a promising path to faithful uncertainty quantification in deep learning.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Probabilistic Methods (eg, variational inference, causal inference, Gaussian processes)