A Dynamical Systems-Inspired Pruning Strategy for Addressing Oversmoothing in Graph Attention Networks

Biswadeep Chakraborty, Harshit Kumar, Saibal Mukhopadhyay

TL;DR: DYNAMO-GAT prevents deep neural networks from losing node distinctiveness by using dynamical systems theory to selectively prune attention weights, achieving superior accuracy with better computational efficiency than existing methods.

Abstract: Graph Neural Networks (GNNs) face a critical limitation known as oversmoothing, where increasing network depth leads to homogenized node representations, severely compromising their expressiveness. We present a novel dynamical systems perspective on this challenge, revealing oversmoothing as an emergent property of GNNs' convergence to low-dimensional attractor states. Based on this insight, we introduce **DYNAMO-GAT**, which combines noise-driven covariance analysis with Anti-Hebbian learning to dynamically prune attention weights, effectively preserving distinct attractor states. We provide theoretical guarantees for DYNAMO-GAT's effectiveness and demonstrate its superior performance on benchmark datasets, consistently outperforming existing methods while requiring fewer computational resources. This work establishes a fundamental connection between dynamical systems theory and GNN behavior, providing both theoretical insights and practical solutions for deep graph learning.

Lay Summary:

Graph Neural Networks (GNNs), a type of AI designed for network data like social or molecular networks, suffer from a problem called "oversmoothing." As these AIs get deeper to learn more complex patterns, they often start seeing all data points (nodes) as increasingly similar, losing crucial distinctions and harming performance. We introduce a fresh perspective, viewing this issue through the lens of dynamical systems – like a physical system settling into an overly simple, uniform state.

Based on this, we developed DYNAMO-GAT. This method cleverly "prunes" or removes connections within the network that contribute most to this uniformity. It identifies these connections by observing how node similarities evolve when a little noise is introduced, a bit like tracking how ripples spread to understand water's flow. This dynamic pruning prevents the AI from collapsing into a single, uninformative state, allowing it to maintain diverse and meaningful insights. Our experiments show DYNAMO-GAT significantly outperforms current methods, achieving higher accuracy in deep networks while being more computationally efficient. This work provides a new way to understand and solve oversmoothing, paving the way for more powerful AI on complex graphs.

Primary Area: Deep Learning->Graph Neural Networks

Keywords: graph neural networks, oversmoothing, dynamical systems, attention mechanisms, pruning optimization, graph attention networks, spectral analysis, anti-hebbian learning, node representation

Submission Number: 9094