Positional Information Can Emerge Through Causal Attention Making Nearby Token Embeddings Similar Even Without Positional Encodings

Chunsheng Zuo, Pavel Guerzhoy, Michael Guerzhoy

Keywords: mechanistic interpretability, position information encoding, nope

TL;DR: We propose a new theory for how causal transformers store positional information

Abstract: Transformers with causal attention can solve tasks that require positional information without using positional encodings. In this work, we propose and investigate a new hypothesis about how positional information can be stored without using explicit positional encoding. We observe that nearby embeddings are more similar to each other than faraway embeddings, allowing the transformer to potentially reconstruct the positions of tokens. We show that this pattern can occur in both the trained and the randomly initialized Transformer models with causal attention and no positional encodings over a common range of hyperparameters.

Track: Main track

Submitted Paper: No

Published Paper: No

Submission Number: 79