Graph Unitary Message Passing

TMLR Paper6903 Authors

Abstract: Unitarity has emerged as a fundamental principle for efficient learning of deep neural networks, from parameter initialization to advanced optimizers, proven effective in various fields, including RNN, CNN, Transformer and Muon optimizer. However, imposing unitarity to the parameters is not enough to improve learning efficiency of graph neural networks (GNNs) due to the instability arising from the graph structure through the message passing mechanism. This data-dependent inefficiency, also known as oversquashing or oversmoothing problems, causes information from distant nodes to decay or node representation to become indistinguishable as the number of layers increases. Motivated by the success of unitarity in stabilizing neural network training, we propose a new graph-learning paradigm called Graph Unitary Message Passing (GUMP) to improve graph learning efficiency by applying unitary adjacency matrices for message passing. GUMP introduces a graph transformation algorithm that equips general graphs with unitary adjacency matrices while preserving original connectivity, and implements Newton-Schulz iteration for efficient unitary projection. Extensive experiments demonstrate that GUMP achieves significant performance improvements over vanilla message passing methods across various graph learning tasks.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Mathurin_Massias1

Submission Number: 6903