The Locality and Symmetry of Positional Encodings

Lihu Chen, Gael Varoquaux, Fabian M. Suchanek

Submission Type: Regular Long Paper

Submission Track: Language Modeling and Analysis of Language Models

Keywords: Positional Encodings, Sentence Representations, Pre-trained Language Models

TL;DR: Positional Encodings in Setence Representations

Abstract:

Positional Encodings (PEs) are used to inject word-order information into
transformer-based language models. While they can significantly enhance
the quality of sentence representations, their specific contribution to
language models is not fully understood,
especially given recent findings that various positional encodings are insensitive to word order.
In this work, we conduct a systematic study of positional encodings in \textbf{Bidirectional Masked Language Models} (BERT-style)
, which complements existing work in three aspects: (1) We uncover the core function of PEs by identifying two common properties, Locality and Symmetry;
(2) We show that the two properties are closely correlated with the performances of downstream tasks;
(3) We quantify the weakness of current PEs by introducing two new probing tasks, on which current PEs perform poorly.
We believe that these results are the basis for developing better PEs for transformer-based language models.

Submission Number: 3245