Revisiting Residual Connections for Neural Structure Learning
Abstract: Recent studies reveal that deep representation learning models without proper regularization can suffer from the dimensional collapse issue, i.e., representation vectors span over a lower dimensional space. In the domain of graph deep representation learning, the phenomenon that the node representations are indistinguishable and even shrink to a constant vector is called oversmoothing. Based on the analysis of the rank of node representations, we find that representation oversmoothing and dimensional collapse are highly related to each other for deep graph neural networks (GNNs), and the oversmoothing problem can be interpreted by the dimensional collapse of the representation matrix. Then, to address the dimensional collapse and the triggered oversmoothing in deep graph neural networks, we first find vanilla residual connections and contrastive learning producing sub-optimal outcomes by ignoring the structural information of graph data. Motivated by this, we propose a novel graph neural network named GearGNN to address the oversmoothing issue from the perspective of addressing dimensional collapse in two folds. Specifically, in GearGNN, we design a topology-preserving residual connection for graph neural networks to force the low-rank of hidden representations close to the full-rank input features. Also, we propose the structure-guided contrastive loss to ensure only close neighbors who share similar local connections can have similar representations. Empirical experiments on multiple real-world datasets demonstrate that GearGNN outperforms state-of-the-art deep graph representation baseline algorithms.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Vlad_Niculae2
Submission Number: 2036
Loading