Dangers of Bayesian Model Averaging under Covariate ShiftDownload PDF

21 May 2021, 20:48 (edited 21 Jan 2022)NeurIPS 2021 PosterReaders: Everyone
  • Keywords: Bayesian, Bayesian neural networks, neural networks, covariate shift, out-of-distribution generalization
  • TL;DR: We demonstrate, explain, and remedy poor performance of Bayesian neural networks under covariate shift.
  • Abstract: Approximate Bayesian inference for neural networks is considered a robust alternative to standard training, often providing good performance on out-of-distribution data. However, Bayesian neural networks (BNNs) with high-fidelity approximate inference via full-batch Hamiltonian Monte Carlo achieve poor generalization under covariate shift, even underperforming classical estimation. We explain this surprising result, showing how a Bayesian model average can in fact be problematic under covariate shift, particularly in cases where linear dependencies in the input features cause a lack of posterior contraction. We additionally show why the same issue does not affect many approximate inference procedures, or classical maximum a-posteriori (MAP) training. Finally, we propose novel priors that improve the robustness of BNNs to many sources of covariate shift.
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
  • Code: https://github.com/izmailovpavel/bnn_covariate_shift
14 Replies