Text Attributed Graph Node Classification Using Sheaf Neural Networks and Large Language Models

ZhongYu Li, Geng Zhao

Keywords: Text Attributed Graph, Sheaf Neural Networks, Large Language Models

Abstract:

Text-Attributed Graphs (TAGs) seamlessly integrate textual data with graph structures, presenting unique challenges and opportunities for jointly modeling text and graph information. Recent advancements in Large Language Models (LLMs) have significantly enhanced the generative and predictive capabilities of text modeling. However, existing graph models often fall short in capturing intricate node relationships, as their edge representations are typically limited to scalar values.

In this paper, we introduce \model, a novel method that encodes rich and complex relational information between nodes as edge vectors. During the message-passing phase, \model aggregates both neighbor node representations and edge vectors to update the central node's representation, eliminating the need to fine-tune the LLMs on the text-attributed graph.

Specifically, for a given TAG, \model is trained to minimize the prediction errors of the LLM in forecasting the next word in node text sequences. Furthermore, we enhance \model's performance by incorporating prompt-based fine-tuning techniques. Once trained, \model can be seamlessly adapted to various downstream tasks.

Extensive node classification experiments across multiple domains demonstrate that \model consistently achieves state-of-the-art performance, validating its effectiveness in capturing complex relationships within TAGs. Additionally, we conduct ablation studies and scalability analyses to ensure the robustness and applicability of our approach.

Primary Area: learning on graphs and other geometries & topologies

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4234