Heteroscedastic Variational Last Layers

James Harrison, John Willes, Paul Brunzema, Jasper Snoek

Keywords: Heteroscedasticity, Bayesian deep learning, variational inference

TL;DR: We introduce a scalable and performant Bayesian deep learning approach for heteroscedastic settings.

Abstract: We improve the performance of Variational Bayesian Last Layer (VBLL) networks by better modeling aleatoric noise. In particular, we (1) Introduce t-VBLL layers, which perform variational inference for the noise covariance, and (2) Introduce Het-VBLL, a Bayesian last layer scheme to model heteroscedastic noise. These methods are based on novel, analytically tractable evidence lower bounds. We show that these novel design elements extend the capabilities of VBLLs at minimal additional cost, and substantially improve performance.

Submission Number: 31