Scalable Multi-Source Pre-training for Graph Neural Networks

Mingkai Lin, Wenzhong Li, Xiaobin Hong, Sanglu Lu

Abstract: Graph Neural Networks (GNNs) have been shown as powerful tools in various scenarios, such as multimodal and multimedia. A fundamental approach, pre-training on available graphs and subsequently transferring the acquired knowledge to optimize downstream tasks with limited labels, was widely exploited to mitigate the demand for extensive labeled training data. However, previous works commonly assumed that pre-training and fine-tuning occur in the same or closely related domains that share similar feature/label spaces and graph distributions. A limitation is that for each individual graph without accessible pre-training data, a GNN must be trained from scratch, imposing high training overhead and hindering the ability of generalization. In this paper, we address the \emph{GNN multi-domain pre-training problem}, which intends to pre-train a transferable GNN model from heterogeneous multi-source graph domains and then apply it in an unseen one with minor fine-tuning costs. To this end, we propose a sca\underline{LA}ble \underline{M}ulti-source \underline{P}re-training (LAMP) method. For pre-training, LAMP presents a graph dual-distillation approach to distill massive knowledge from various graph domains to form synthetic homogeneous graphs. Simultaneously, high-level meta-knowledge from the synthetic graphs is extracted to train the GNN model, whose capability can be adjusted according to target graph contexts through a co-training modulation architecture. For fine-tuning, LAMP respectively aligns the target graph distribution, graph context, and graph task with the pretext so that the downstream task in the unseen domain can be reshaped to leverage the transferable knowledge efficiently. Extensive experiments on four real-world graph domain datasets demonstrate the superiority of LAMP, showcasing notable improvements in various downstream graph learning tasks. Our codes are publicly available on GitHub.

Primary Subject Area: [Content] Multimodal Fusion

Secondary Subject Area: [Engagement] Multimedia Search and Recommendation

Relevance To Conference: Graphs offer a flexible tool for representing various structural and unstructured data, especially in multimodal and multimedia scenarios. In this work, we propose the framework LAMP to pre-train a GNN model from heterogeneous multi-source graph domains and then apply it to an unseen graph domain with minor fine-tuning costs. Therefore, through this work, by employing Graph Neural Networks (GNNs) across heterogeneous domains, researchers can leverage the inherent ability of GNNs to handle complex relationships and dependencies within and across different modalities, facilitating tasks such as multimodal fusion, cross-modal retrieval, or content-based recommendation systems.

Submission Number: 933