An Attention-Based Approach to Accelerating Sequence Generative Adversarial Nets

Minglei Gao, Sai Zhang, Xiaowang Zhang, Zhiyong Feng

Abstract: Automatic text generation is widely used in dialogue systems, machine translation and other fields. Sequence Generative Adversarial Network (SeqGAN) has achieved good performance in text generation tasks. Due to the discriminator can only evaluate the finished text, and cannot provide other valid information to the generator. When evaluating a single word, the Monte Carlo algorithm is mainly used to generate a complete text. This process requires a huge computational cost. As the length of the text increases, the time to obtain rewards will increase significantly. For text, different words have different effects on semantics, and keywords determine the final expression of semantics. Evaluation of the importance of each word is particularly critical. In this paper, we propose a new framework called AttGAN. We allow the discriminator to provide more features to the generator. Specifically, we add an attention layer to the new discriminator. The attention score is used as the basic reward so that the discriminator can calculate the reward value of each word through only one evaluation without multiple sampling by the generator. And to meet the requirements of valid reward, we further process the attention score. Our large number of experiments on synthetic data and tests on dialogue systems show that AttGAN can minimize computational costs and generate high-quality text. Furthermore, it also has a good performance in the generation of lengthy text.