LLM4GCL: CAN LARGE LANGUAGE MODEL EM-POWER GRAPH CONTRASTIVE LEARNING?

Yi Fang, Xiaqiang Tang, Luyang Shang, Dongzhe Fan, Daochen Zha, Qiaoyu Tan

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: GNN, Graph Contrastive Learning, LLM

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: This paper explores several feasible pipelines that utilize LLM to enhance the performance of GCL

Abstract: Graph contrastive learning (GCL) has made significant strides in pre-training graph neural networks (GNNs) without requiring human annotations. Previous GCL efforts have primarily concentrated on augmenting graphs, assuming the node features are pre-embedded. However, many real-world graphs contain textual node attributes (e.g., citation network), known as text-attributed graphs (TAGs). The existing GCL methods often simply convert the textual attributes into numerical features using shallow or heuristic methods like skip-gram and bag-of-words, which cannot capture the semantic nuances and general knowledge embedded in natural language. Motivated by the exceptional capabilities of large language models (LLMs), like ChatGPT, in comprehending text, in this work, we delve into the realm of GCL on TAGs in the era of LLMs, which we term LLM4GCL. We explore two potential pipelines: \textit{LLM-as-GraphAugmentor} and \textit{LLM-as-TextEncoder}. The former aims to directly leverage LLMs to conduct augmentations at the feature and structure levels through prompts. The latter attempts to employ LLMs to encode nodes' textual attributes into embedding vectors. Building on these two pipelines, we conduct comprehensive and systematic studies on six benchmark datasets, exploring various feasible designs. The results show the promise of LLM4GCL in enhancing the performance of state-of-the-art GCL methods. Our code and dataset will be publicly released upon acceptance.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6944