Relative Positional Encoding Family via Unitary Transformation

Zhen Qin, Weixuan Sun, Kaiyue Lu, Hui Deng, Dongxu Li, XiaoDong Han, Yuchao Dai, Lingpeng Kong, Yiran Zhong

Keywords: Linear Transformer, Relative positional encoding, Unitary transformation

TL;DR: Introducing unitary relative positional encoding, a principal design for relative position encoding, applicable inclusively for linear and vanilla transformer.

Abstract: Relative position encoding is widely used in vanilla and linear transformers to represent positional information. However, the existing encoding methods of a vanilla transformer are not always directly applicable to a linear transformer, because the latter requires a decomposition of the query and key representations into separate kernel functions. Nevertheless, principles to design encoding methods suitable for linear transformers remain under-studied. In this work, we put together a variety of existing encoding methods under a canonical form and further propose a family of relative positional encodings via unitary transformation. Our formulation leads to a principled framework that can be used to develop new relative positional encoding methods that preserve linear space-time complexity. Equipping with different parameters, the proposed unitary relative positional encoding family (URPE) derives effective encoding for various applications. Experiments show that compared with existing encoding methods, unitary encoding achieves competitive performance on language modeling and various challenging downstream tasks, such as machine translation and text classification. In the meantime, it highlights a general paradigm to design broadly more relative positional encoding methods, applicable inclusively to linear and vanilla transformers.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning