Mixtures of Locally Bounded Langevin dynamics for Bayesian Model Averaging

Kilian Zepf, Tareen Dawood, Aasa Feragen, Ender Konukoglu

Abstract: Properties of probability distributions change when going from low to high dimensions, to the extent that they exhibit counterintuitive behavior. Gaussian distributions intuitively illustrate a well-known effect of moving to higher dimensions, namely that the typical set almost surely does not contain the mean, which is the distribution's most probable point. This can be problematic in Bayesian Deep Learning, as the samples drawn from the high-dimensional posterior distribution are often used as Monte Carlo samples to estimate the integral of the predictive distribution. Here, the predictive distribution will reflect the behavior of the samples and, therefore, of the typical set. For instance, we cannot expect to sample networks close to the maximum a posteriori estimate after fitting a Gaussian approximation to the posterior using the Laplace method. In this paper, we introduce a method that aims to mitigate this typicality problem in high dimensions by sampling from the posterior with Langevin dynamics on a restricted support enforced by a reflective boundary condition. We demonstrate how this leads to improved posterior estimates by illustrating its capacity for fine-grained out-of-distribution (OOD) ranking on the Morpho-MNIST dataset.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

The authors found that some changes were required upon acceptance:

1. Due to an error in Figure 6, the Thick and Thin results for the ResNet were erroneously the same in the Figure, but this has been corrected.
2. The authors decide to include previous feedback from Reviewer uEJt regarding the epsilon size, which has now been updated in Section 3.4.

Code: https://github.com/tareend/TMLR_lbMALA.git

Assigned Action Editor: ~Michele_Caprio1

Submission Number: 5860