Is Pre-training Truly Better Than Meta-Learning?

Brando Miranda, Saumya Goyal, Patrick Yu, Yu-Xiong Wang, Sanmi Koyejo

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: meta-learning, pre-training, data centric machine learning, machine learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We show that when considering the formal diversity of a data set -- one can show MAML beats pre-training

Abstract:

n the context of few-shot learning, it is currently believed that a fixed pre-trained (PT) model, along with fine-tuning the final layer during evaluation, outperforms standard meta-learning algorithms.
We re-evaluate these claims under an in-depth empirical examination of an extensive set of formally diverse datasets and compare PT to Model Agnostic Meta-Learning (MAML).
Unlike previous work, we emphasize a fair comparison by using: the same architecture, the same optimizer, and all models trained to convergence.
Crucially, we use a more rigorous statistical tool -- the effect size (Cohen's d) -- to determine the practical significance of the difference between a model trained with PT vs. a MAML.
We then use a previously proposed metric -- the diversity coefficient -- to compute the average formal diversity of a dataset.
Using this analysis, we demonstrate the following:
1. when the formal diversity of a data set is low, PT beats MAML on average and
2. when the formal diversity is high, MAML beats PT on average.
The caveat is that the magnitude of the average difference between a PT vs. MAML using the effect size is low (according to classical statistical thresholds) -- less than 0.2.
Nevertheless, this observation is contrary to the currently held belief that a pre-trained model is always better than a meta-learning model.
Our extensive experiments consider 21 few-shot learning benchmarks, including the large-scale few-shot learning dataset Meta-Data set.
We also show no significant difference between a MAML model vs. a PT model with GPT-2 on Openwebtext.
We, therefore, conclude that a pre-trained model does not always beat a meta-learned model and that the formal diversity of a dataset is a driving factor.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6534