Information Theoretic Meta Learning with Gaussian Processes

Michalis Titsias, Sotirios Nikoloutsopoulos, Alexandre Galashov

Keywords: Meta Learning, Information Bottleneck, Gaussian Processes, Few-shot learning, Variational Inference

Abstract: We formulate meta learning using information theoretic concepts such as mutual information and the information bottleneck. The idea is to learn a stochastic representation or encoding of the task description, given by a training or support set, that is highly informative about predicting the validation set. By making use of variational approximations to the mutual information, we derive a general and tractable framework for meta learning. We particularly develop new memory-based meta learning algorithms based on Gaussian processes and derive extensions that combine memory and gradient-based meta learning. We demonstrate our method on few-shot regression and classification by using standard benchmarks such as Omniglot, mini-Imagenet and Augmented Omniglot.

One-sentence Summary: Meta learning using information bottleneck and combination of memory and gradient-based techniques using Gaussian processes.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2009.03228/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=Gc5Drbr45H