Is Attention Better Than Matrix Decomposition?

Zhengyang Geng, Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, Zhouchen Lin

Keywords: attention models, matrix decomposition, computer vision

Abstract: As an essential ingredient of modern deep learning, attention mechanism, especially self-attention, plays a vital role in the global correlation discovery. However, is hand-crafted attention irreplaceable when modeling the global context? Our intriguing finding is that self-attention is not better than the matrix decomposition~(MD) model developed 20 years ago regarding the performance and computational cost for encoding the long-distance dependencies. We model the global context issue as a low-rank completion problem and show that its optimization algorithms can help design global information blocks. This paper then proposes a series of Hamburgers, in which we employ the optimization algorithms for solving MDs to factorize the input representations into sub-matrices and reconstruct a low-rank embedding. Hamburgers with different MDs can perform favorably against the popular global context module self-attention when carefully coping with gradients back-propagated through MDs. Comprehensive experiments are conducted in the vision tasks where it is crucial to learn the global context, including semantic segmentation and image generation, demonstrating significant improvements over self-attention and its variants. Code is available at https://github.com/Gsunshine/Enjoy-Hamburger.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Code: [ Gsunshine/Enjoy-Hamburger](https://github.com/Gsunshine/Enjoy-Hamburger) + [ 1 community implementation](https://paperswithcode.com/paper/?openreview=1FvkSpWosOl)

Data: [ADE20K](https://paperswithcode.com/dataset/ade20k), [ImageNet](https://paperswithcode.com/dataset/imagenet), [PASCAL Context](https://paperswithcode.com/dataset/pascal-context), [PASCAL VOC 2012 test](https://paperswithcode.com/dataset/pascal-voc-2012-test)

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:2109.04553/code)