Do Bayesian Neural Networks Actually Behave Like Bayesian Models?

Gábor Pituk, Vik Shirvaikar, Tom Rainforth

Abstract: We empirically investigate how well popular approximate inference algorithms for Bayesian Neural Networks (BNNs) respect the theoretical properties of Bayesian belief updating. We find strong evidence on synthetic regression and real-world image classification tasks that common BNN algorithms such as variational inference, Laplace approximation, SWAG, and SGLD fail to update in a consistent manner, forget about old data under sequential updates, and violate the predictive coherence properties that would be expected of Bayesian methods. These observed behaviors imply that care should be taken when treating BNNs as true Bayesian models, particularly when using them beyond static prediction settings, such as for active, continual, or transfer learning.

Lay Summary:

Bayesian Neural Networks (BNNs) are used to help machine learning models express uncertainty. This is useful in tasks like decision making, where knowing what the model is unsure about can matter as much as the prediction itself.

We test whether BNNs behave in ways that are intuitively important: remembering past data, updating sensibly when new data arrives, and not changing predictions without new information. Across a range of experiments, we find that widely used BNN methods often violate these basic expectations. They can forget earlier data, produce inconsistent updates, and behave as if they've seen new data when they haven't.

These behaviors raise concerns about using BNNs in real-world situations where learning over time or from limited data is critical. While BNNs can perform well in static tasks, our findings suggest the need for caution, and for further research into improving how these models handle uncertainty and learning.

Primary Area: Probabilistic Methods->Bayesian Models and Methods

Keywords: Bayesian Neural Networks, BNNs, Bayesian Deep Learning

Submission Number: 13257