Sliced Recursive Transformer

Zhiqiang Shen, Zechun Liu, Eric Xing

Keywords: Recursive Operation, Vision Transformer, Efficient Model, Approximating Self-Attention, Sliced Group Self-Attention

Abstract: We present a neat yet effective recursive operation on vision transformers that can improve parameter utilization without involving additional parameters. This is achieved by sharing weights across depth of transformer networks. The proposed method can obtain a substantial gain (~2%) simply using naive recursive operation, requires no special or sophisticated knowledge for designing principles of networks, and introduces minimum computational overhead to the training procedure. To reduce the additional computation caused by recursive operation while maintaining the superior accuracy, we propose an approximating method through multiple sliced group self-attentions across recursive layers which can reduce the cost consumption by 10~30% with minimal performance loss. We call our model Sliced Recursive Transformer (SReT), which is compatible with a broad range of other designs for efficient vision transformers. Our best model establishes significant improvement on ImageNet over state-of-the-art methods while containing fewer parameters. The proposed sliced recursive operation allows us to build a transformer with more than 100 or even 1000 layers effortlessly under a still small size (13~15M), to avoid difficulties in optimization when the model size is too large. The flexible scalability has shown great potential for scaling up and constructing extremely deep and large dimensionality vision transformers.

One-sentence Summary: This work introduces a sliced recursive vision transformer for the efficient transformer design.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2111.05297/code)