Adversarial-Based Ensemble Feature Knowledge Distillation

Mingwen Shao, Shunhang Li, Zilu Peng, Yuantao Sun

Abstract: Existing knowledge distillation methods usually consider class probabilities or directly align features, which ignore the rich information contained in high-level feature maps. In this paper, we propose a novel adversarial-based ensemble feature knowledge distillation (AEFKD) that utilizes probabilistic information and high-level feature information for enhanced online knowledge distillation. Specifically, we first obtain the ensemble features of more efficient sub-networks through the feature ensemble module. And then distinguish the feature maps distributions of sub-networks and ensemble features through discriminators. Finally, each sub-network learns the distribution of ensemble features through deceiving its corresponding discriminator. Our proposed AEFKD not only considers the differences between category probabilities, but also utilizes adversarial learning to transfer the knowledge of the feature maps. The AEFKD is evaluated on a widely used benchmark dataset with various network architectures. Extensive experiments show that AEFKD outperforms state-of-the-art online knowledge distillation methods.

External IDs:dblp:journals/npl/ShaoLPS23