GeoPE:A Unified Geometric Positional Embedding for Structured Tensors

Yupu Yao, Bowen Yang

Keywords: Positional Embedding, Vision Transformer, RoPE

TL;DR: This paper introduces GeoPE, which extends 1D RoPE to higher dimensions using quaternions and Lie algebra, achieving consistently strong performance on various 2D and 3D vision tasks while maintaining excellent generalization to unseen resolutions.

Abstract: Standard Vision Transformers flatten 2D images into 1D sequences, disrupting the natural spatial topology. While Rotary Positional Embedding (RoPE) excels in 1D, it inherits this limitation, often treating spatially distant patches (e.g., at row edges) as sequence neighbors. Existing 2D approaches typically treat spatial axes independently, failing to decouple this false sequential proximity from true spatial distance. To restore the 2D spatial manifold, we introduce Geometric Positional Embedding (GeoPE), a framework that extends rotations to 3D Euclidean space using quaternions. To overcome non-commutativity and ensure symmetry, GeoPE constructs a unified rotational operator by computing the geometric mean in the Lie algebra. This creates a geometrically coupled encoding that effectively separates spatial dimensions. Extensive experiments on image classification, object detection, and 3D semantic segmentation demonstrate that GeoPE consistently outperforms existing 2D RoPE variants and significantly enhances shape bias, confirming its ability to capture true geometric structure.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 19045