Informed weight initialization of Graph Neural Networks and its effect on Oversmoothing

Dimitrios Kelesis, Dimitris Fotakis, Georgios Paliouras

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Graph Neural Networks, Weight initialization, Oversmoothing

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

In this work, we generalize the ideas of Kaiming initialization to Graph Neural Networks (GNNs) and propose a new initialization scheme that addresses the problem of oversmoothing. GNNs are typically initialized using methods, that have been designed for other types of Neural Networks, such as Xavier or Kaiming initialization. Such methods ignore the underlying topology of the graph. In this work, propose a new initialization method, called G-Init, which takes into account (a) the variance of signals flowing forward, (b) the gradients flowing backward
in the network, and (c) the effect of graph convolution, which tends to smooth node representations and lead to the problem of oversmoothing. Oversmoothing is an inherent problem of GNNs, which appears when their depth increases, making node representations indistinguishable. We show that in deep GNNs, G-Init reduces oversmoothing and enables deep architectures. We also verify the theoretical results experimentally.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6007