A Text GAN for Language Generation with Non-Autoregressive Generator

Fei Huang, Jian Guan, Pei Ke, Qihan Guo, Xiaoyan Zhu, Minlie Huang

Keywords: language generation, non-autoregressive text generation, generative adversarial networks, GANs, latent variable

Abstract: Despite the great success of Generative Adversarial Networks (GANs) in generating high-quality images, GANs for text generation still face two major challenges: first, most text GANs are unstable in training mainly due to ineffective optimization of the generator, and they heavily rely on maximum likelihood pretraining; second, most text GANs adopt autoregressive generators without latent variables, which largely limits the ability to learn latent representations for natural language text. In this paper, we propose a novel text GAN, named NAGAN, which incorporates a non-autoregressive generator with latent variables. The non-autoregressive generator can be effectively trained with gradient-based methods and free of pretraining. The latent variables facilitate representation learning for text generation applications. Experiments show that our model is competitive comparing with existing text GANs in unconditional text generation, and it outperforms existing methods on sentence manipulation in latent space and unsupervised text decipherment.

One-sentence Summary: We propose a text GAN with a non-autoregressive generator, which can be effectively trained with gradient-based method from scratch and applied to text generation applications that require latent variables.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=puYOXv7IMp