Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs
Keywords: Knowledge Distillation, Graph Neural Networks, multilayer perceptrons, Edge-free, Structure-aware
TL;DR: We propose a Prototype-Guided Knowledge Distillation~(PGKD) method, which does not require graph edges as extra input~(edge-free setting) yet learns structure-aware MLPs from GNN teachers.
Abstract: Distilling high-accuracy Graph Neural Networks (GNNs) to low-latency multilayer perceptrons (MLPs) on graph tasks has become a hot research topic. However, conventional MLPs almost exclusively rely on the graph nodes and fail to effectively capture the graph structural information. Previous methods address this issue by processing graph edges into extra inputs for MLPs, but such graph structures may be unavailable for various scenarios. To this end, we propose Prototype-Guided Knowledge Distillation (PGKD), which does not require graph edges (edge-free setting) yet learns structure-aware MLPs. Our insight is to distill graph structural information from GNNs. Specifically, we first analyze the impact of graph structures on GNN teachers, and then design two losses based on prototypes to distill such information from GNNs to MLPs. Experimental results on popular graph benchmarks demonstrate the effectiveness and robustness of the proposed PGKD.
Supplementary Material: zip
Primary Area: general machine learning (i.e., none of the above)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 7075
Loading