Learning the RoPEs: Better 2D and 3D Position Encodings with STRING

Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Kumar Avinava Dubey, Ayzaan Wahid, Sumeet Singh, René Wagner, Tianli Ding, Chuyuan Fu, Arunkumar Byravan, Jake Varley, Alexey A. Gritsenko, Matthias Minderer, Dmitry Kalashnikov, Jonathan Tompson et al. (2 additional authors not shown)

TL;DR: This paper introduces STRING: a general class of separable translation-invariant position encodings for Transformers (extending Rotary Position Encodings), and applies them in a wide range of 2D and 3D scenarios, in particular in Robotics.

Abstract:

We introduce $\textbf{STRING}$: Separable Translationally Invariant Position Encodings. STRING extends Rotary Position Encodings, a recently proposed and widely used algorithm in large language models, via a unifying theoretical framework. Importantly, STRING still provides $\textbf{exact}$ translation invariance, including token coordinates of arbitrary dimensionality, whilst maintaining a low computational footprint. These properties are especially important in robotics, where efficient 3D token representation is key.
We integrate STRING into Vision Transformers with RGB(-D) inputs (color plus optional depth), showing substantial gains, e.g. in open-vocabulary object detection and for robotics controllers.
We complement our experiments with a rigorous mathematical analysis, proving the universality of our methods. Videos of STRING-based robotics controllers can be found here: https://sites.google.com/view/string-robotics.

Lay Summary: This paper introduces STRING, a new and improved method for AI to understand the position of items, especially in 2D images and 3D scenes. Current AI models (Transformers) grasp content but struggle with order or location. STRING builds upon a popular method called RoPE but is more general and better suited for multi-dimensional data. It retains RoPE's key benefits—encoding each item's position independently ("separability") and focusing on relative distances ("translational invariance")—while being more powerful. The paper proves STRING is theoretically the most comprehensive approach of its kind under certain conditions. Crucially, it delivers significant performance gains in practical applications like object detection and robotics control, where efficiently representing 2D/3D information is vital. In short, STRING helps AI "see" and understand spatial arrangements more effectively.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Primary Area: Deep Learning->Algorithms

Keywords: position encodings for Transformers, RoPE, Robotics, translation-invariance, Lie algebra

Flagged For Ethics Review: true

Submission Number: 5392