Prompt-based Node Feature Extractor for Few-shot Learning on Text-Attributed Graph

Xuanwen Huang, Kaiqiao Han, Dezheng Bao, Yang Yang, Qi Zhu

Keywords: Text-attributed graph; graph neural network; language model

Abstract:

Text-attributed Graphs (TAGs) are commonly found in the real world, such as social networks and citation networks, which consist of nodes represented by textual descriptions.
Currently, mainstream machine learning methods on TAGs involve a two-stage modeling approach: (1) unsupervised node feature extraction with pre-trained language models (PLMs); and (2) supervised learning using Graph Neural Networks (GNNs).
However, we observe that these representations, which have undergone large-scale pre-training, do not significantly improve performance with limited amount of training samples.
The main issue is that existing methods have not effectively integrated information from the graph and downstream tasks simultaneously.
First, G-Prompt introduces a learnable GNN layer (i.e., adaptor) at the end of PLMs, which is fine-tuned to better capture the masked tokens considering graph neighborhood information.
After the adapter is trained, G-Prompt incorporates task-specific prompts to obtain interpretable node representations for the downstream task.
Our experiment results demonstrate that our proposed method outperforms current state-of-the-art (SOTA) methods on few-shot node classification. More importantly, in zero-shot settings, the G-Prompt embeddings can not only provide better task interpretability than vanilla PLMs
but also achieve comparable performance with fully-supervised baselines.

Submission Number: 14283