Enhancing Graph Transformers with Spectral Guidance in Attention

ICLR 2026 Conference Submission18777 Authors

Keywords: Graph Learning, Graph Transformer, Graph Spectral Theory

TL;DR: We propose SGA-Former, a novel graph Transformer that first leverages spectral priors as inductive bias to explicitly facilitate selective attention learning and and achieve superior performance across diverse graph tasks.

Abstract: Existing Graph Transformers often overlook the limitations of self-attention mechanism without inductive bias. The pure self-attention tends to aggregate features from unrelated nodes and misalign attention with graph structures, leading to suboptimal modeling of relational dependencies. Moreover, operating solely in the spatial domain, self-attention underutilizes graph spectral components that correspond to more detailed and comprehensive relational patterns. To address the above issues, we propose the Spectral-Guided Attention Graph Transformer (SGA-Former), which introduces rich structural priors from the graph spectral domain to guide attention learning. Specifically, we design two Spectral Relation Metrics as attention bias, which capture complementary low and high-frequency structural patterns. To leverage these priors, we develop the Spectral-Guided Attention Enhancer (SGA-Enhancer), which filters redundant attention scores and emphasizes important node relationships based on the spectral metrics. Incorporating SGA-Enhancer, SGA-Former builds dual-branch Spectral Attention Layers that jointly utilize both spectral views, enabling more balanced and structure-aware attention learning. Extensive experiments show that SGA-Former consistently achieves superior performance across a wide range of graph learning tasks.

Primary Area: learning on graphs and other geometries & topologies

Submission Number: 18777