Invariance Learning in Deep Neural Networks with Differentiable Laplace ApproximationsDownload PDF

Published: 31 Oct 2022, Last Modified: 05 Jan 2023NeurIPS 2022 AcceptReaders: Everyone
Keywords: Bayesian deep learning, Laplace approximation, invariance learning, Bayesian model selection
TL;DR: We learn invariances present in the data for deep neural networks without supervision or validation data using gradient-based Bayesian model selection.
Abstract: Data augmentation is commonly applied to improve performance of deep learning by enforcing the knowledge that certain transformations on the input preserve the output. Currently, the data augmentation parameters are chosen by human effort and costly cross-validation, which makes it cumbersome to apply to new datasets. We develop a convenient gradient-based method for selecting the data augmentation without validation data during training of a deep neural network. Our approach relies on phrasing data augmentation as an invariance in the prior distribution on the functions of a neural network, which allows us to learn it using Bayesian model selection. This has been shown to work in Gaussian processes, but not yet for deep neural networks. We propose a differentiable Kronecker-factored Laplace approximation to the marginal likelihood as our objective, which can be optimised without human supervision or validation data. We show that our method can successfully recover invariances present in the data, and that this improves generalisation and data efficiency on image datasets.
Supplementary Material: pdf
11 Replies