EMPIRICAL PRIORS FOR BAYESIAN NEURAL NETWORKS VIA WEIGHT PRUNING

Heejung Shin, Jiwoo Kim, Seulki Lee, Hyondong Oh

Keywords: Bayesian neural network, variational inference, pruning, empirical bayes, prior

TL;DR: We propose SPIN, a method that uses network pruning to set informative priors for Bayesian Neural Networks, improving performance across various architectures and datasets.

Abstract: Designing informative priors for Bayesian neural networks (BNNs) remains a fundamental challenge, yet it plays a crucial role in determining both model performance and robustness. While numerous studies have explored effective strategies for prior selection, the suitable choice of prior distribution is often architecture-dependent and requires extensive time to determine. To address this, we propose Sparsity-Informed priors for Bayesian neural networks, SPIN, a simple method that empirically determines both the prior mean and variance of Gaussian priors: weights that survive pruning are considered important and are assigned low-variance Gaussian priors centered at their post-pruning values, while pruned weights are treated as less informative and given high-variance, zero-mean Gaussian priors. Our empirical results demonstrate that SPIN enhances performance across diverse architectures and datasets. Furthermore, we discuss how this prior design contributes to improved performance in Bayesian neural networks.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 9167