Meta-Learning Bayesian Neural Network Priors Based on PAC-Bayesian Theory

Jonas Rothfuss, Martin Josifoski, Andreas Krause

Keywords: meta-learning, life-long learning, transfer, bayesian neural networks, prior, few-shot learning, pac-bayes, generalization bound

Abstract:

Bayesian deep learning is a promising approach towards improved uncertainty quantification and sample efficiency.
Due to their complex parameter space, choosing informative priors for Bayesian Neural Networks (BNNs) is challenging. Thus, often a naive, zero-centered Gaussian is used, resulting both in bad generalization and poor uncertainty estimates when training data is scarce. In contrast, meta-learning aims to extract such prior knowledge from a set of related learning tasks. We propose a principled and scalable algorithm for meta-learning BNN priors based on PAC-Bayesian bounds. Whereas previous approaches require optimizing the prior and multiple variational posteriors in an interdependent manner, our method does not rely on difficult nested optimization problems and is agnostic to the variational inference method in use. Our experiments show that the proposed method is not only computationally more efficient but also yields better predictions and uncertainty estimates when compared to previous meta-learning methods and BNNs with standard priors.

One-sentence Summary: A principled and scalable meta-learning algorithm for Bayesian neural network priors based on PAC-Bayesian learning thory.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=HvY9o0Hf0t