Variational Bayesian Last Layers

James Harrison, John Willes, Jasper Snoek

Keywords: bayesian deep learning, variational methods, bayesian last layers

TL;DR: We provide a simple, cheap, and deterministic Bayesian training procedure for neural network last layers

Abstract: We introduce a deterministic variational formulation for training Bayesian last layer neural networks. This yields a sampling-free, single-forward pass objective that effectively improves network uncertainty representation. Our variational Bayesian last layer can be trained and evaluated inexpensively, with only quadratic complexity in last layer width, and is thus (nearly) computationally free to add to existing architectures.