A Low-Rank Perspective on Oversmoothing in Graph Neural Networks
Keywords: graph neural network, oversmoothing
TL;DR: We advocate the use of the effective rank as a more robust and informative metric to quantify oversmoothing
Abstract: Oversmoothing is a fundamental challenge in graph neural networks (GNNs): as the number of layers increases, the node embeddings become progressively similar, leading to smoothened representations. This phenomenon often results in a sharp performance drop after only a few layers, significantly limiting the depth of GNNs. Traditionally, oversmoothing has been quantified using norm-based energy metrics, such as the Dirichlet energy, which measures the norm of differences between neighbouring node features. These metrics decay only when the embeddings converge to either a rank-one or an all-zero representation as depth increases.
However, we argue that these metrics offer an overly simplistic view and fail to reliably capture oversmoothing in realistic scenarios, such as when network weights are unbounded, graph adjacency matrices are not stochastic, or activation functions are highly non-homogeneous (e.g. $\tanh$). In such cases, the embeddings may not collapse to a rank-one representation, and norm-based energy metrics fail to detect a drop in representational quality. Instead, we propose measuring the effective rank of the representations, which provides a more nuanced understanding of oversmoothing. Our findings reveal that a significant drop in effective rank corresponds closely with performance degradation, even in cases where energy metrics remain unchanged. Extensive evaluations across diverse graph architectures demonstrate that rank-based metrics consistently capture oversmoothing, unlike energy-based approaches, which often fail.
Submission Number: 20
Loading