Learning to Transfer Learn

Linchao Zhu, Sercan O. Arik, Yi Yang, Tomas Pfister

TL;DR: We propose learning to transfer learn (L2TL) to improve transfer learning on a target dataset by judicious extraction of information from a source dataset.

Abstract: We propose learning to transfer learn (L2TL) to improve transfer learning on a target dataset by judicious extraction of information from a source dataset. L2TL considers joint optimization of vastly-shared weights between models for source and target tasks, and employs adaptive weights for scaling of constituent losses. The adaptation of the weights is based on reinforcement learning, guided with a performance metric on the target validation set. We demonstrate state-of-the-art performance of L2TL given fixed models, consistently outperforming fine-tuning baselines on various datasets. In the regimes of small-scale target datasets and significant label mismatch between source and target datasets, L2TL outperforms previous work by an even larger margin.

Keywords: transfer learning, adaptive training

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1908.11406/code)