Fine-Grained Learning Behavior-Oriented Knowledge Distillation for Graph Neural Networks

Kang Liu, Zhenhua Huang, Chang-Dong Wang, BeiBei Gao, Yunwen Chen

Abstract: Knowledge distillation (KD), as an effective compression technology, is used to reduce the resource consumption of Graph Neural Networks (GNNs) and facilitate their deployment on resource-constrained devices. Numerous studies exist on GNNs distillation, however, the impacts of knowledge complexity and differences in learning behavior between teachers and students on distillation efficiency remain underexplored. We propose a knowledge distillation method for fine-grained learning behavior (FLB), comprising two main components: Feature Knowledge Decoupling (FKD) and Teacher Learning Behavior Guidance (TLBG). Specifically, FKD decouples the intermediate-layer features of the student network into two types: Teacher-Related Features (TRF) and Downstream Features (DF), enhancing knowledge comprehension and learning efficiency by guiding the student to simultaneously focus on these features. TLBG maps the teacher model's learning behaviors to provide reliable guidance for correcting deviations in student learning. Extensive experiments across eight datasets and 12 baseline frameworks demonstrate that FLB significantly enhances the performance and robustness of student GNNs within the original framework.