Rethinking the Smoothness of Node Features Learned by Graph Convolutional Networks

Shih-Hsin Wang, Justin Baker, Cory D Hauck, Bao Wang

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: graph neural networks, activation function, smoothness of node features

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We develop a fine-grained analysis of how ReLU and leaky ReLU affect the smoothness of node features learned by GCN

Abstract: It has been proved that graph convolutional layers (GCLs) using ReLU or leaky ReLU activation function smooth node features. Such a smoothing process is beneficial for node classification using a few GCLs. However, deep graph convolutional networks (GCNs) tend to learn homogeneous node feature vectors over the graph, making nodes indistinguishable. In this paper, we develop a new understanding of the smoothness of node features learned by GCNs by establishing a fine-grained analysis of how ReLU or leaky ReLU affects the smoothness of its input vectors. First, we establish a geometric relationship between the input and output of ReLU or leaky ReLU. Then we show that if one ignores the magnitude of the feature vectors, ReLU and leaky ReLU smooth their input feature vectors, echoing existing theory. We further show that taking the magnitude of feature vectors into account, ReLU and leaky ReLU can increase, decrease, or preserve the smoothness of their input vectors. Our theory informs the design of a simple yet effective approach to let GCN learn node features with a desired smoothness that improves its empirical performance for graph node classification.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4062