Group-Equivariant Transformers Without Positional Encoding

Byungjin Kim, Minsu Cho

Keywords: equivariant, invariant, group-equivariant, self-attention, transformer, group-equivariant convolution, group-equivariant self-attention

TL;DR: We propose an effective group-equivariant transformer without positional encoding, replacing point-wise MLPs with group-equivariant convolutions to act as both a group mixer and an implicit positional encoding.

Abstract: Self-attention is a permutation-equivariant operator in its basic form and can further extend to achieve equivariance for a specific symmetry group by incorporating group-invariant positional encoding. In this work, we propose an effective group-equivariant transformer without positional encoding. Instead of injecting group-invariant position encoding to the transformer, we replace point-wise MLPs with group-equivariant convolutions that act as both a group mixer and an implicit positional encoding. This allows to reduce the group of self-attention to translation only while preserving group equivariance, resulting in less computation and memory. Our strategy not only retains dynamic long-range interactions of transformers but also incorporates the static effective kernel learning of convolution, resulting in a significant accuracy gain. We also find that adopting a group-equivariant convolution stem and a translation-equivariant pooling further improves the performance. The proposed method sets a new state of the art in standard benchmarks, outperforming the existing group-equivariant transformers by a large margin.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning