Meta-learning Transferable Representations with a Single Target Domain

Hong Liu, Jeff Z. HaoChen, Colin Wei, Tengyu Ma

Keywords: transfer learning, fine-tuning, supervised transfer learning

Abstract: Recent works found that fine-tuning and joint training---two popular approaches for transfer learning---do not always improve accuracy on downstream tasks. First, we aim to understand more about when and why fine-tuning and joint training can be suboptimal or even harmful for transfer learning. We design semi-synthetic datasets where the source task can be solved by either source-specific features or transferable features. We observe that (1) pre-training may not have incentive to learn transferable features and (2) joint training may simultaneously learn source-specific features and overfit to the target. Second, to improve over fine-tuning and joint training, we propose Meta Representation Learning MeRLin to learn transferable features. MeRLin meta-learns representations by ensuring that a head fit on top of the representations with target training data also performs well on target validation data. We also prove that MeRLin recovers the target ground-truth model with a quadratic neural net parameterization and a source distribution that contains both transferable and source-specific features. On the same distribution, pre-training and joint training provably fail to learn transferable features. MeRLin empirically outperforms previous state-of-the-art transfer learning algorithms on various real-world vision and NLP transfer learning benchmarks.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We introduce meta representation learning to overcome the limitations of fine-tuning and joint training in a transfer learning setting with a single target domain.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2011.01418/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=d245bbroV