LinkGPT: Teaching Large Language Models To Predict Missing Links
Keywords: link prediction; large language model
Abstract: Large Language Models (LLMs) have shown promising results on various language and vision tasks. Recently, there has been growing interest in applying LLMs to graph-based tasks, particularly on Text-Attributed Graphs (TAGs). However, most studies have focused on node classification, while the use of LLMs for link prediction (LP) remains understudied. In this work, we propose a new task on LLMs, where the objective is to leverage LLMs to predict missing links between nodes in a graph. This task evaluates an LLM’s ability to reason over structured data and infer new facts based on learned patterns. This new task poses two key challenges: (1) How to effectively integrate pairwise structural information into the LLMs, which is known to be crucial for LP performance, and (2) how to solve the computational bottleneck when teaching LLMs to perform LP. To address these challenges, we propose LinkGPT, the first LLM-based training and inference framework specifically designed for LP tasks on homogeneous TAGs. To enhance the LLM's ability to understand the underlying structure, we design a two-stage instruction tuning approach where the first stage finetunes the pairwise encoder, projector, and node projector, and the second stage further finetunes the LLMs to predict links. To address the efficiency challenges at inference time, we introduce a retrieval-reranking scheme and investigate three LLM-based retrieval methods. Extensive experiments show that LinkGPT can achieve state-of-the-art performance on real-world graphs and superior generalization in zero-shot and few-shot learning, surpassing existing benchmarks. At inference time, it can achieve 10x speedup while maintaining high LP accuracy.
Submission Number: 90
Loading