RiemannGFM: Learning a Graph Foundation Model from Structural Geometry
Track: Graph algorithms and modeling for the Web
Keywords: Graph Neural Network, Foundation Model, Riemannian Geometry
TL;DR: This work opens a new opportunity to build graph foundation model with a shared structural vocabulary of graph domain, and we further connect the vocabulary to Riemannian geometry.
Abstract: The foundation model has heralded a new era in artificial intelligence, pretraining a single model to offer cross-domain transferability on different datasets. Graphs are omnipresent non-Euclidean structures, ranging from recommender systems to biochemical structures. Graph neural networks excel at learning graph data, but often lack the generalization capacity. Hence, graph foundation model is drawing increasing attention, and recent efforts have been made to leverage Large Language Models, encouraged by the remarkable success of GPT-4. On the one hand, existing studies primarily focus on text-attributed graphs, while a wider range of real graphs do not contain fruitful textual attributes. On the other hand, the sequential graph description tailored for the Large Language Model neglects the structural complexity, which is a predominant characteristic of the graph. Such limitations motivate an important question: Can we go beyond Large Language Model, and pretrain a universal model to learn the structural knowledge for any graph? The answer in the language or vision domain is a shared vocabulary. We observe the fact that there also exist shared substructures underlying the graph domain, and thereby open the new opportunity of graph foundation model with structure vocabulary (by which any graph can be constructed). The key innovation of this paper is the discovery of a simple yet effective structural vocabulary of trees and cycles, and we explore its inherent connection to Riemannian geometry. Herein, we present a universal pretraining model, RiemannGFM, with geometric contrastive learning. Concretely, we first construct a novel product bundle to incorporate the diverse geometries of the vocabulary. On this constructed space, we stack Riemannian layers where the structural vocabulary, regardless of specific graph, is learnt in Riemannian manifold. This offers the shared structural knowledge for cross-domain transferability, and node encoding is generated in the tangent bundle for arbitrary input graph. Empirical results show the superiority of RiemannGFM on a diversity of real graphs.
Submission Number: 982
Loading