Efficiently Warmstarting MCMC for BNNs

David Rundel, Emanuel Sommer, Bernd Bischl, David Rügamer, Matthias Feurer

Keywords: Bayesian Neural Networks, MCMC, Computational Efficiency, Warmstarting, Performance Predictors

TL;DR: We analyze the dynamics of warmstarting MCMC for BNNs and propose novel strategies that leverage performance predictors and adaptive termination criteria.

Abstract:

Markov Chain Monte Carlo (MCMC) algorithms are widely regarded as the gold standard for approximate inference in Bayesian neural networks (BNNs). However, they remain computationally expensive and prone to inefficiencies, such as dying samplers, frequently leading to substantial waste of computational resources. While prior work has presented warmstarting techniques as an effective
method to mitigate these inefficiencies, we provide a more comprehensive empirical analysis of how initializations of samplers affect their behavior. Based on various experiments examining the dynamics of warmstarting MCMC, we propose novel warmstarting strategies that leverage performance predictors and adaptive termination criteria to achieve better-performing, yet more cost-efficient, models. In numerical experiments, we demonstrate that this approach provides a practical pathway to more resource-efficient approximate inference in BNNs.

Submission Number: 61