SpRePE: A Spherical Geometry-Aware Position Embedding for Vision Transformers

Qilong Jia, Yi Xiao, Wei Xue

Keywords: Vision Transformer, Position Embedding, Geometry-Aware Modeling, Spherical Data Processing

TL;DR: We propose SpRePE, a spherical position embedding scheme for Transformers. It encodes absolute positions on the sphere via a Householder matrix and incorporates an explicit spherical relative-position term into the attention formulation.

Abstract:

Position embedding (PE) is a key mechanism that breaks the permutation symmetry of tokens in Transformer, introducing a spatial inductive bias that enables attention to model locality, distances, and directional relations.
Spherical data arise in many scientific domains, most notably in astronomy and meteorology, where Vision Transformers is increasingly adopted for the ability to capture long-range dependencies. However, conventional PEs are designed for linear sequences and cannot faithfully capture the sphere’s non-Euclidean geometry.
Furthermore, existing designs for encoding spherical positional information rely on additional network modules or specialized network architectures, which introduce extra parameters and computational overhead.
These limitations motivate a geometry-aware and efficient embedding scheme that fully exploits spherical structure to advance Transformer-based modeling on the sphere.
We introduce \textbf{Spherical Reflection Position Embedding (SpRePE)}, a lightweight method efficiently leveraging spherical positional information for Vision Transformer.
SpRePE encodes the absolute position on the sphere using a Householder matrix and incorporates the explicit relative position dependency into the attention formulation, achieving both high computational efficiency and high accuracy without requiring substantial additional parameters and modifications to the overall model architecture.
We evaluate SpRePE on representative tasks, including spherical image classification and global weather forecasting. SpRePE consistently outperforms well-known baselines including APE, RPE, ALiBi and RoPE.
These results indicate that SpRePE offers an efficient and broadly applicable position embedding scheme for Transformer models on the sphere.

Primary Area: learning on graphs and other geometries & topologies

Submission Number: 5045