Multi-modal Prompt Learning Empowers Graph Neural Networks with Semantic Knowledge

Zihao Li, Lecheng Zheng, Bowen Jin, Dongqi Fu, Baoyu Jing, Yikun Ban, Jingrui He, Jiawei Han

Keywords: Graph Foundation Model, Multi-modal Prompt Learning, Graph Neural Network, Language Models, Contrastive Learning

Abstract: While great success has been achieved in building generalizable language models, three fundamental issues hinder GNN-based graph foundation models: the scarcity of labeled data, different levels of downstream tasks, and the conceptual gaps between domains. In depth, though the labels of real graphs are associated with semantic information, most graph learning frameworks ignore it by turning semantic labels into numerical labels. In this work, to address these issues, we present a new paradigm that leverages the text modality to align downstream tasks and data with any pre-trained GNN given only a few semantically labeled samples. Our paradigm embeds the graphs directly in the same space as the LLM by learning both graph prompts and text prompts simultaneously. To accomplish this, we improve state-of-the-art graph prompt method based on our theoretical findings. Then, we propose the first multi-modal prompt learning approach for exploiting the knowledge in pre-trained models. Notably, in our paradigm, the pre-trained GNN and the LLM are kept frozen, so the number of learnable parameters is much smaller than fine-tuning any pre-trained model. Through extensive experiments on real-world datasets, we demonstrate the superior performance of our paradigm in few-shot, multi-task-level, and cross-domain settings. Moreover, we build the first zero-shot classification prototype that can generalize GNNs to unseen classes. The code is provided in the supplementary materials.

Primary Area: foundation or frontier models, including LLMs

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5477