STransGAN: An Empirical Study on Transformer in GANs

Rui Xu, Xiangyu Xu, Kai Chen, Bolei Zhou, Chen Change Loy

Keywords: GAN, Transformer, generative models

Abstract: Transformer becomes prevalent in computer vision, especially for high-level vision tasks. However, deploying Transformer in the generative adversarial network (GAN) framework is still an open yet challenging problem. In this paper, we conduct a comprehensive empirical study to investigate the intrinsic properties of Transformer in GAN for high-fidelity image synthesis. Our analysis highlights the importance of feature locality in image generation. We first investigate the effective ways to implement local attention. We then examine the influence of residual connections in self-attention layers and propose a novel way to reduce their negative impacts on learning discriminators and conditional generators. Our study leads to a new design of Transformers in GAN, a convolutional neural network (CNN)-free generator termed as STrans-G, which achieves competitive results in both unconditional and conditional image generations. The Transformer-based discriminator, STrans-D, also significantly reduces its gap against the CNN-based discriminators. Models and code will be publicly available.

One-sentence Summary: We conduct a comprehensive empirical study to investigate the intrinsic properties of Transformer in GAN for high-fidelity image synthesis.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2110.13107/code)