Towards Foundation Models on Graphs: An Analysis on Cross-Dataset Transfer of Pretrained GNNs

Fabrizio Frasca, Fabian Jogl, Moshe Eliasof, Matan Ostrovsky, Carola-Bibiane Schönlieb, Thomas Gärtner, Haggai Maron

Keywords: Graph Neural Networks, Foundation Models, Transfer Learning, GNNs, Graph Foundation Models

TL;DR: We study the extent to which pretrained Graph Neural Networks can be applied across datasets, analyzing the impact of feature information, pretraining datasets, and the amount of downstream data.

Abstract:

To develop a preliminary understanding towards Graph Foundation Models, we study the extent to which pretrained Graph Neural Networks can be applied across datasets, an effort requiring to be agnostic to dataset-specific features and their encodings. We build upon a purely structural pretraining approach and propose an extension to capture feature information while still being feature-agnostic. We evaluate pretrained models on downstream tasks for varying amounts of training samples and choices of pretraining datasets. Our preliminary results indicate that embeddings from pretrained models improve generalization only with enough downstream data points and in a degree which depends on the quantity and properties of pretraining data. Feature information can lead to improvements, but
currently requires some similarities between pretraining and downstream feature spaces.

Submission Number: 59