The Effect of diversity in Meta-Learning

Ramnath Kumar, Tristan Deleu, Yoshua Bengio

Keywords: Meta-Learning, Few-shot learning

Abstract: Few-shot learning aims to learn representations that can tackle novel tasks given a small number of examples. Recent studies show that task distribution plays a vital role in the performance of the model. Conventional wisdom is that task diversity should improve the performance of meta-learning. In this work, we find evidence to the contrary; we study different task distributions on a myriad of models and datasets to evaluate the effect of task diversity on meta-learning algorithms. For this experiment, we train on multiple datasets, and with three broad classes of meta-learning models - Metric-based (i.e., Protonet, Matching Networks), Optimization-based (i.e., MAML, Reptile, and MetaOptNet), and Bayesian meta-learning models (i.e., CNAPs). Our experiments demonstrate that the effect of task diversity on all these algorithms follows a similar trend, and task diversity does not seem to offer any benefits to the learning of the model. Furthermore, we also demonstrate that even a handful of tasks, repeated over multiple batches, would be sufficient to achieve a performance similar to uniform sampling and draws into question the need for additional tasks to create better models.

One-sentence Summary: Analysis of different task sampling schemes for meta-learning to understand the effect of task diversity in meta-learning

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:2201.11775/code)