Towards Principled Graph Transformers

Luis Müller, Daniel Kusuma, Blai Bonet, Christopher Morris

Keywords: graph transformers, expressivity, Weisfeiler and Leman, Weisfeiler and Lehman

TL;DR: We study graph transformers that are both theoretically grounded in the Weisfeiler-Leman hierarchy as well as perform comparative with state-of-the-art on graph learning benchmarks.

Abstract: The expressive power of graph learning architectures based on the $k$-dimensional Weisfeiler-Leman ($k$-WL) hierarchy is well understood. However, such architectures often fail to deliver solid predictive performance on real-world tasks, limiting their practical impact. In contrast, global attention-based models such as graph transformers demonstrate strong performance in practice, but comparing their expressive power with the $k$-WL hierarchy remains challenging, particularly since these architectures rely on positional or structural encodings for their expressivity and predictive performance. To address this, we show that the recently proposed Edge Transformer, a global attention model operating on node pairs instead of nodes, has 3-WL expressive power when provided with the right tokenization. Empirically, we demonstrate that the Edge Transformer surpasses other theoretically aligned architectures regarding predictive performance while not relying on positional or structural encodings.

Primary Area: Graph neural networks

Submission Number: 12766