Feature Matters: A Stage-by-Stage Approach for Task Independent Knowledge Transfer

Mengya Gao, Yujun Shen, Quanquan Li, Liang Wan, Xiaoou Tang

Abstract: Convolutional Neural Networks (CNNs) become deeper and deeper in recent years, making the study of model acceleration imperative. It is a common practice to employ a shallow network, called student, to learn from a deep one, which is termed as teacher. Prior work made many attempts to transfer different types of knowledge from teacher to student, however, there are two problems remaining unsolved. Firstly, the knowledge used by existing methods is highly dependent on task and dataset, limiting their applications. Secondly, there lacks an effective training scheme for the transfer process, leading to degradation of performance. In this work, we argue that feature is the most important knowledge from teacher. It is sufficient for student to just learn good features regardless of the target task. From this discovery, we further present an efficient learning strategy to mimic features stage by stage. Extensive experiments demonstrate the importance of features and show that the proposed approach significantly narrows down the gap between student and teacher, outperforming the state-of-the-art methods.

Keywords: knowledge transfer, task independent, feature transfer, stage-by-stage

TL;DR: This paper proposes to transfer knowledge from deep model to shallow one by mimicking features stage by stage.