Contrastive Enhanced Knowledge Distillation for Learning MLPs on GNNs

Wei Hu, Juhua Pu, Xiaolan Tang, Xingwu Liu

Abstract: In recent years, graph neural networks (GNNs) have emerged as a promising approach for classifying non-Euclidean structural data. However, the practical implementation of GNNs faces challenges related to their limited scalability due to the presence of multi-hop data dependencies. In order to tackle this issue, existing methods have employed teacher GNNs to generate labels, which are then used to train multilayer perceptrons (MLPs) based solely on node features, without considering any structural information. However, these methods primarily focus on the soft labels generated by the teacher GNNs, leading to suboptimal performance because they disregard significant features of the teacher GNNs. Additionally, since the structural information of the graph is omitted in the input of MLPs, it can be more susceptible to the influence of erroneous features. To address these limitations, this paper proposes a novel framework that incorporates a new distillation strategy to integrate soft feature similarity into MLPs, while also utilizing contrastive learning to enhance the training of student MLPs. Our model has accuracy, averaged over seven public datasets, 5.21% higher than state-of-the-art (SOTA) methods, and even 2.49% higher than teacher GNNs over five standard scaled datasets. At the same time, its inference time is only 1.17% of comparable GNNs.

External IDs:dblp:journals/npl/HuPTL25