Wasserstein Generalization Bound for Few-Shot Learning

Phani raj Chinnalingu, Rajarshi Banerjee

Keywords: Few shot learning, Generalization

TL;DR: We use properties of wassertein distance to give a tight bound for few shot learning, specifically prototypical networks.

Abstract:

In the absence of large quantities of annotated data, few shot learning is used to train neural
networks that make predictions based on similarities between datapoints. To better under-
stand how models would behave when presented with unfamiliar data, research on gen-
eralization bounds have revealed some important properties about deep neural networks.
However, when extended to the domain of few shot learning it often yields loose bounds
since it does not take into the account the nature and methodology of few shot learning.
We propose a novel stochastic generalization bound for prototypical neural networks by
constructing a Wasserstein sphere centered around the distribution of weight matrices. We
show that by applying concentration inequalities on the distribution of weight matrices in
the Wasserstein sphere stricter generalization bounds can be obtained. Comparison with
previous generalization bounds shows the efficacy of our approach and to our knowledge
this is the first bound that makes use of Wasserstein distance to give a measure of general-
izability of deep neural networks

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning