Analyzing the effect of residual connections to oversmoothing in graph neural networks

Dimitrios Kelesis, Dimitris Fotakis, Georgios Paliouras

Abstract: The performance of Graph Neural Networks (GNNs) diminishes as their depth increases. That is mainly attributed to oversmoothing, which leads to similar node representations through repeated graph convolutions. To enable deep GNNs, several approaches have been proposed, among which the use of residual connections. Residual connections have proven effective in benchmark datasets, but the way in which they improve the performance of deep GNNs has not been fully studied. We show that residual connections force the model to focus on the local neighborhood of graph nodes, making the GNN equivalent to the sum of shallow GCNs. We explain theoretically why this is the case and verify the theoretical results experimentally. However, our findings raise the question of whether residual connections are helpful in cases where deep networks are necessary. We assess this experimentally, in two situations: (a) in the presence of the “cold start" problem, i.e. when there is no feature information about unlabeled nodes; and (b) in a new synthetic dataset of controllable long-interactions. These experiments highlight the drawbacks of GNNs using residual connections, while showing that simpler methods can be more effective.