Bayes Always Wins the Lottery in Monte Carlo

ICLR 2026 Conference Submission18884 Authors

Keywords: Bayesian Neural Networks, Neural Network Pruning, Hamiltonian Monte Carlo, Lottery Ticket Hypothesis

Abstract: Most contemporary neural networks suffer from huge model sizes requiring prohibitive storage and computational resources for training, making their use difficult on edge devices. Neural network pruning aims to address this problem. The lottery ticket hypothesis is a sparse pruning method that can reduce the network size significantly with minimal accuracy loss. However, the original initialization sample is needed in order for the "winning ticket" to train to the same accuracy after pruning. We present a novel approach utilizing Hamiltonian Monte Carlo to always win the lottery by training Bayesian neural networks with lottery-ticket generated pruning masks from any initialization. Our first key finding is to establish a generalized framework for training lottery ticket pruned networks, independent of specific initialization samples, with a Bayesian-based theoretical grounding containing convergence guarantees that ensure the optimal initialization distribution is found. Second is that networks trained using this framework achieve predictive performance equivalent to or exceeding that of networks initialized with the lottery ticket initialization sample. Finally, we investigate whether stochastic gradient based Bayesian methods can achieve similar performance as Hamiltonian Monte Carlo. Result highlights include that on LENET300-100 networks on CIFAR-10 using Hamiltonian Monte Carlo, we observed a best-case accuracy improvement of 5\% over random initialization samples and 3\% over the original lottery-ticket initialization sample, highlighting the capabilities of Bayesian methods for training pruned networks.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 18884