Dual teachers for self-knowledge distillation

Zheng Li, Xiang Li, Lingfeng Yang, Renjie Song, Jian Yang, Zhigeng Pan

Abstract: Highlights•To our best knowledge, we are the first work to jointly conduct historical and structural self-knowledge distillation in a simple and unified framework, where they demonstrate complementary advantages and mutual benefits.•We demonstrate that an extremely lightweight design of auxiliary branches outperforms the previous heavy counterparts in both efficiency and accuracy.•Extensive experiments prove the effectiveness of our proposed framework on three datasets: CIFAR-100, ImageNet-2012, and PASCAL VOC.