PolyFormer: Scalable Graph Transformer via Polynomial Attention

Jiahong Ma, Mingguo He, Zhewei Wei

Primary Area: learning on graphs and other geometries & topologies

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Graph Transformer, Graph Filter, Graph Neural Network

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: The paper presents PolyFormer, a scalable Graph Transformer using PolyAttn for node-level tasks, handling graphs up to 100 million nodes.

Abstract: Graph Transformers have demonstrated superior performance in graph representation learning. However, many current methods focus on attention mechanisms between node pairs, limiting their scalability and expressiveness on node-level tasks. While the recent NAGphormer attempts to address scalability by employing node tokens in conjunction with vanilla multi-head self-attention, these tokens, which are designed in the spatial domain, suffer from restricted expressiveness. On the other front, some approaches have explored encoding eigenvalues or eigenvectors in the spectral domain to boost expressiveness, but these methods incur significant computational overhead due to the requirement for eigendecomposition. To overcome these limitations, we first introduce node tokens using various polynomial bases in the spectral domain. Then, we propose a tailored polynomial attention mechanism, PolyAttn, which serves as a node-wise graph filter and offers powerful representation capabilities. Building on PolyAttn, we present PolyFormer, a graph Transformer model specifically engineered for node-level tasks, offering a desirable balance between scalability and expressiveness. Extensive experiments demonstrate that our proposed methods excel at learning arbitrary node-wise filters, showing superior performance on both homophilic and heterophilic graphs, and handling graphs containing up to 100 million nodes.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8779