On Structural Expressive Power of Graph Transformers

Wenhao Zhu, Tianyu Wen, Guojie Song, Liang Wang, Bo Zheng, Xi Wang

Keywords: graph representation learning, graph isomorphism testing, graph Transformer

Abstract: Graph Transformer (GT) has recently received wide attention in the research community with its outstanding performance, yet its structural expressive power has not been well analyzed. Inspired by the connections between Weisfeiler-Lehman (WL) graph isomorphism test and graph neural network (GNN), we introduce \textbf{GT test}, a generalized graph isomorphism test algorithm as a powerful theoretical tool for exploring the structural discriminative power of graph Transformers. We theoretically prove that the GT test is an expressivity upper bound on a wide range of graph Transformers, and the representational power of GT test can be approximated by a simple Transformer network arbitrarily under certain conditions. With the GT test, we show how graph Transformers' expressive power is determined by the design of structural encodings, and present conditions that make the expressivity of graph Transformers beyond WL test and GNNs. Moreover, motivated by the popular shortest path distance encoding, we follow the theory-oriented principles and develop a provably stronger structural encoding method, Shortest Path Induced Subgraph (\textit{SPIS}) encoding. Our theoretical findings provide a novel and practical paradigm for investigating the expressive power of graph Transformers, and extensive synthetic and real-world experiments empirically verify the strengths of our proposed methods.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning

TL;DR: Investigating the expressive power of graph Transformers.

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:2305.13987/code)