Depth Scaling in Graph Neural Networks: Understanding the Flat Curve Behavior

Diana Gomes, Kyriakos Efthymiadis, Ann Nowe, Peter Vrancx

Abstract: Training deep Graph Neural Networks (GNNs) has proved to be a challenging task. A key goal of many new GNN architectures is to enable the depth scaling seen in other types of deep learning models. However, unlike deep learning methods in other domains, deep GNNs do not show significant performance boosts when compared to their shallow counterparts (resulting in a flat curve of performance over depth). In this work, we investigate some of the reasons why this goal of depth still eludes GNN researchers. We also question the effectiveness of current methods to train deep GNNs and show evidence of different types of pathological behavior in these networks. Our results suggest that current approaches hide the problems with deep GNNs rather than solve them, as current deep GNNs are only as discriminative as their respective shallow versions.

Submission Length: Regular submission (no more than 12 pages of main content)

Code: https://github.com/dsg95/gnn-depth-eval

Assigned Action Editor: ~Lechao_Xiao2

Submission Number: 2131