GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphs

Yichuan Li, Kaize Ding, Kyumin Lee

Submission Type: Regular Long Paper

Submission Track: Information Retrieval and Text Mining

Submission Track 2: Language Modeling and Analysis of Language Models

Keywords: textual attributed graph, text rich network, representation learning, graph neural network, language model

TL;DR: GRENADE combines language models & graph networks via self-supervised algorithms, excelling in text-attributed graph representation.

Abstract:

Self-supervised representation learning on text-attributed graphs, which aims to create expressive and generalizable representations for various downstream tasks, has received increasing research attention lately. However, existing methods either struggle to capture the full extent of structural context information or rely on task-specific training labels, which largely hampers their effectiveness and generalizability in practice. To solve the problem of self-supervised representation learning on text-attributed graphs, we develop a novel Graph-Centric Language model -- GRENADE.
Specifically, GRENADE harnesses the synergy of both pre-trained language model and graph neural network by optimizing with two specialized self-supervised learning algorithms: graph-centric contrastive learning and graph-centric knowledge alignment. The proposed graph-centric self-supervised learning algorithms effectively help GRENADE to capture informative textual semantics as well as structural context information on text-attributed graphs. Through extensive experiments, GRENADE shows its superiority over state-of-the-art methods.

Submission Number: 2611