Wavelet-Induced Rotary Encodings: RoPE Meets Graphs

Isaac Reid, Arijit Sehanobish, Cederik Höfs, Bruno Kacper Mlodozeniec, Leonhard Vulpius, Federico Barbero, Adrian Weller, Krzysztof Marcin Choromanski, Richard E. Turner, Petar Veličković

Keywords: rotary position encodings, graphs, spectra, transformers, attention, performers

TL;DR: A RoPE-style position encoding for graph-structured data

Abstract: We introduce WIRE: Wavelet-Induced Rotary Encodings. WIRE extends Rotary Position Encodings (RoPE), a popular algorithm in LLMs and ViTs, to graph-structured data. We demonstrate that WIRE is more general than RoPE, recovering the latter in the special case of grid graphs. WIRE also enjoys a host of desirable theoretical properties, including equivariance under node ordering permutation, compatibility with linear attention, and (under select assumptions) asymptotic dependence on graph resistive distance. We test WIRE on a range of synthetic and real-world tasks, including identifying monochromatic subgraphs, semantic segmentation of point clouds, and more standard graph benchmarks. We find it to be effective in settings where the underlying graph structure is important.

Primary Area: learning on graphs and other geometries & topologies

Submission Number: 6347