Unlocking Graph Structure Learning with Tree-Guided Large Language Models

Zhihan Zhang, Xunkai Li, Zhu Lei, Guang Zeng, Rong-Hua Li, Guoren Wang

Keywords: Graph Neural Networks, Graph Structure Learning, Large Language Models

TL;DR: The first attempt to redefine the paradigm of graph structure learning in the era of LLMs, achieving superior performance while enjoying high running efficiency.

Abstract: Recently, the emergence of large language models (LLMs) has prompted researchers to integrate language descriptions into graphs, aiming to enhance model encoding capabilities from a data-centric perspective. This graph representation is called text-attributed graphs (TAGs). A review of prior advancements highlights that graph structure learning (GSL) is a pivotal technique for improving data utility, making it highly relevant to efficient TAG learning. However, most GSL methods are tailored for traditional graphs without textual information, underscoring the necessity of developing a new GSL paradigm. Despite clear motivations, it remains challenging: (1) How can we define a reasonable optimization objective for GSL in the era of LLMs, considering the massive parameters in LLMs? (2) How can we design an efficient model architecture that enables seamless integration of LLMs for this optimization objective? For Question 1, we reformulate existing GSL optimization objectives as a tree optimization framework, shifting the focus from obtaining a well-trained edge predictor to a language-aware tree sampler. For Question 2, we propose decoupled and training-free model design principles for LLM integration, shifting the focus from computation-intensive fine-tuning to more efficient inference. Based on this, we propose Large Language and Tree Assistant (LLaTA), which leverages tree-based LLM in-context learning to enhance the understanding of topology and text, enabling reliable inference and generating improved graph structure. Extensive experiments on 11 datasets demonstrate that LLaTA enjoys flexibility—incorporated with any backbone; scalability—outperforms other LLM-based GSL methods; and effectiveness—achieving SOTA predictive performance across a variety of datasets from different domains.

Primary Area: learning on graphs and other geometries & topologies

Submission Number: 12519