Graph Generative Pre-trained Transformer

Xiaohui Chen, Yinkai Wang, Jiaxing He, Yuanqi Du, Soha Hassoun, Xiaolin Xu, Liping Liu

TL;DR: A novel approach for modeling graph using GPT-like model

Abstract: Graph generation is a critical task in numerous domains, including molecular design and social network analysis, due to its ability to model complex relationships and structured data. While most modern graph generative models utilize adjacency matrix representations, this work revisits an alternative approach that represents graphs as sequences of node set and edge set. We advocate for this approach due to its efficient encoding of graphs and propose a novel representation. Based on this representation, we introduce the Graph Generative Pre-trained Transformer (G2PT), an auto-regressive model that learns graph structures via next-token prediction. To further exploit G2PT's capabilities as a general-purpose foundation model, we explore fine-tuning strategies for two downstream applications: goal-oriented generation and graph property prediction. We conduct extensive experiments across multiple datasets. Results indicate that G2PT achieves superior generative performance on both generic graph and molecule datasets. Furthermore, G2PT exhibits strong adaptability and versatility in downstream tasks from molecular design to property prediction.

Lay Summary:

This paper revisits how to build auto-regressive models from generative pre-trained transformers for graph generation- The key is tokenization. Specifically, we tokenize graphs into node and edge sets. This tokenization differs significantly from commonly used adjacency matrix representation, which greatly saves comp. and memory costs by exploiting sparsity of graphs. Building on this tokenization, we can adopt all existing recipes for generative pre-trained transformers in LLMs.

We further expand the capability of G2PT to goal-oriented generation via post-training techniques: (1) rejection sampling fine-tuning and (2) Reinforcement learning.We also validate the quality of the learned embedding on graph property prediction problems and achieve better or on-par performance with other graph contrastive and generative pre-training methods. In the end, we analyze the scaling behavior of G2PT and find on molecular graph generation, it scales very well with the data and model size and saturates when getting close to 100% validity.

Link To Code: https://github.com/tufts-ml/G2PT

Primary Area: Deep Learning->Generative Models and Autoencoders

Keywords: Graph generation, transformer, graph representation learning

Submission Number: 1417