Efficient Knowledge Distillation from Model Checkpoints

Chaofei Wang, Qisen Yang, Rui Huang, Shiji Song, Gao Huang

Keywords: Deep learning, image classification, intermediate model, knowledge distillation, mutual information.

Abstract: Knowledge distillation is an effective approach to learn compact models (students) with the supervision of large and strong models (teachers). As empirically there exists a strong correlation between the performance of teacher and student models, it is commonly believed that a high performing teacher is preferred. Consequently, practitioners tend to use a well trained network or an ensemble of them as the teacher. In this paper, we observe that an intermediate model, i.e., a checkpoint in the middle of the training procedure, often serves as a better teacher compared to the fully converged model, although the former has much lower accuracy. More surprisingly, a weak snapshot ensemble of several intermediate models from a same training trajectory can outperform a strong ensemble of independently trained and fully converged models, when they are used as teachers. We show that this phenomenon can be partially explained by the information bottleneck principle: the feature representations of intermediate models can have higher mutual information regarding the input, and thus contain more ``dark knowledge'' for effective distillation. We further propose an optimal intermediate teacher selection algorithm based on maximizing the total task-related mutual information. Experiments verify its effectiveness and applicability. Our code is available at https://github.com/LeapLabTHU/CheckpointKD.

TL;DR: This paper explains theoretically and experimentally that appropriate model checkpoints can be more economical and efficient than the fully converged models in knowledge distillation.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2210.06458/code)