Search to Fine-Tune Pre-Trained Graph Neural Networks for Graph-Level Tasks

Zhili Wang, Shimin Di, Lei Chen, Xiaofang Zhou

Abstract: Recently, graph neural networks (GNNs) have shown its unprecedented success in many graph-related tasks. However, GNNs face the label scarcity issue as other neural networks do. Thus, recent efforts try to pre-train GNNs on a large-scale unlabeled graph and adapt the knowledge from the unlabeled graph to the target downstream task. The adaptation is generally achieved by fine-tuning the pre-trained GNNs with a limited number of labeled data. However, current GNNs pre-training works focus more on how to better pre-train a GNN, but ignore the importance of fine-tuning to better leverage the transferred knowledge. Only a few works start to investigate a better fine-tuning strategy for pre-trained GNNs. But their designs either have strong assumptions or overlook the data-aware issue behind various downstream domains. To further boost pre-trained GNNs, we propose to search to fine-tune pre-trained GNNs for graph-level tasks (S2PGNN), which can adaptively design a suitable fine-tuning framework for the given pre-trained GNN and downstream data. Unfortunately, it is a non-trivial task to achieve this goal due to two technical challenges. First is the hardness of fine-tuning space design since there lack a systematic and unified exploration in existing literature. Second is the enormous computational overhead required for discovering suitable fine-tuning strategies from the discrete space. To tackle these challenges, S2PGNN first carefully summarizes a search space of fine-tuning strategies that is suitable for GNNs, which is expressive enough to enable powerful strategies to be searched. Then, S2PGNN integrates an efficient search algorithm to solve the computationally expensive search problem from a discrete and large space. The empirical studies show that S2PGNN can be implemented on the top of 10 famous pre-trained GNNs and consistently improve their performance by 9 % to 17 %. Our code is publicly available at https://github.com/zwangeo/icde2024.