What Happens to the Source Domain in Transfer Learning?

Amal Alnouri, Bilal Alsallakh

Keywords: Transfer Learning

TL;DR: Transfer learning from a supervised model can cause knowledge retained about the source domain to interfere with the target task.

Abstract: We investigate the impact of the source domain in supervised transfer learning, focusing on image classification. In particular, we aim to assess to which extent a fine-tuned model can still recognize the classes of the source domain. Furthermore, we want to understand how this ability impacts the target domain. We demonstrate how the retained knowledge about the old classes in a popular foundational model can interfere with the model’s ability to learn and recognize the new classes. This interference can incur significant implications and highlights an inherent shortcoming of supervised transfer learning.