PatchGT: Transformer over Non-trainable Clusters for Learning Graph Representations

Han Gao, Xu Han, Jiaoyang Huang, Jian-Xun Wang, Liping Liu

Keywords: Graph Neural Networks, Unlearnable spectral clustering method, Transformer on graphs

TL;DR: This paper aims to mitigate the downside of previous learnable/hierarchical clustering methods with a simple spectral clustering methods combining with a transformer on graph patches.

Abstract: Recently the Transformer structure has shown good performances in graph learning tasks. However, these Transformer models directly work on graph nodes and may have difficulties learning high-level information. Inspired by the vision transformer, which applies to image patches, we propose a new Transformer-based graph neural network: Patch Graph Transformer (PatchGT). Unlike previous transformer-based models for learning graph representations, PatchGT learns from non-trainable graph patches, not from nodes directly. It can help save computation and improve the model performance. The key idea is to segment a graph into patches based on spectral clustering without any trainable parameters, with which the model can first use GNN layers to learn patch-level representations and then use Transformer layers to obtain graph-level representations. The architecture leverages the spectral information of graphs and combines the strengths of GNNs and Transformers. Further, We show the limitations of previous hierarchical trainable clusters theoretically and empirically. We also prove the proposed non-trainable spectral clustering method is permutation invariant and can help address the information bottlenecks in the graph. PatchGT achieves higher expressiveness than 1-WL-type GNNs, and the empirical study shows that PatchGT achieves competitive performances on benchmark datasets and provides interpretability to its predictions. The implementation of our algorithm is released at our GitHub repo: https://github.com/tufts-ml/PatchGT.

Type Of Submission: Full paper proceedings track submission (max 9 main pages).

Agreement: Check this if you are okay with being contacted to participate in an anonymous survey.

Type Of Submission: Full paper proceedings track submission.

Software: https://github.com/tufts-ml/PatchGT

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2211.14425/code)