Positional Encodings as Group Representations: A Unified Framework

Derek Lim, Hannah Lawrence, Ningyuan Teresa Huang, Erik Henning Thiede

Keywords: Positional encodings, Group representations, Approximate Equivariance, Transformers, Implicit neural representations

TL;DR: We present a unified framework for positional encodings as group representations and apply it to the group of 2D representations.

Abstract: Positional encodings are ubiquitous as an input featurization tool in language modeling, computer vision, and graph representation learning, enabling neural networks to capture important geometric structure of the input. Traditionally, positional encodings have been defined anew for each data domain. In this work, we reinterpret positional encodings for disparate data types --- including sequences, grids, graphs, and manifolds --- in the unifying framework of group representations. We show how to express existing positional encodings as group representations, and conversely, propose new positional encodings by choosing suitable groups and representations. We validate our framework with experiments on implicit neural representations of images and vector fields, highlighting the practical utility of such positional encodings for encouraging approximate equivariance and capturing geometric structure.

Type Of Submission: Extended Abstract (4 pages, non-archival)

Submission Number: 83