RandAlign: A Parameter-Free Method for Regularizing Graph Convolutional Networks

Haimin ZHANG, Min Xu

Abstract: Studies continually find that message-passing graph convolutional networks suffer from the over- smoothing issue. Basically, the issue of over-smoothing refers to the phenomenon that the learned embeddings for all nodes can become very similar to one another and therefore are uninforma- tive after repeatedly applying message passing iterations. Intuitively, we can expect the generated embeddings become smooth asymptotically layerwisely, that is each layer of graph convolution gen- erates a smoothed version of embeddings as compared to that generated by the previous layer. Based on this intuition, we propose RandAlign, a stochastic regularization method for graph convolutional networks. The idea of RandAlign is to randomly align the learned embedding for each node with that generated by the previous layer using random interpolation in each graph convolution layer. Through alignment, the smoothness of the generated embeddings is explicitly reduced. To better maintain the benefit yielded by the graph convolution, in the alignment step we introduce to first scale the embed- ding of the previous layer to the same norm as the generated embedding and then perform random interpolation for aligning the generated embedding. RndAlign is a parameter-free method and can be directly applied without introducing additional trainable weights or hyper-parameters. We exper- imentally evaluate RandAlign on different graph domain tasks on seven benchmark datasets. The experimental results show that RandAlign is a generic method that improves the generalization per- formance of various graph convolutional network models and also improves the numerical stability of optimization, advancing the state of the art performance for graph representation learning.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission: Revision based on reviewers comments.

Assigned Action Editor: ~Moshe_Eliasof1

Submission Number: 2607