Unlocking the Power of GANs in Non-Autoregressive Text Generation under Weak Conditions

TMLR Paper5171 Authors

Abstract: Non-autoregressive (NAR) models once received great attention from the community, but obtain much less attention in the quest for general artificial intelligence. Our analyses reveal that the convergence problem in existing NAR models trained under Maximum Likelihood Estimation (MLE) is more severe in tasks where input does not provide the definite semantic meaning of the output. These input conditions, which we denote as weak conditions, cover most ``creative'' tasks, so existing NAR models struggle to obtain satisfactory performance in these tasks and are only developed in limited scenarios. This causes existing NAR models to struggle to keep pace with the rapidly evolving demands of diverse and challenging tasks. Different with MLE, which is incompatible with NAR models, Generative Adversarial Networks (GANs) are more suitable for NAR models in terms of theoretical convergence and inference manners. We thus propose an Adversarial Non-autoregressive Transformer (ANT) based on GANs for weak condition tasks. ANT supports two features: 1) Position-Aware Self-Modulation to provide more effective input signals, and 2) Dependency Feed Forward Network to strengthen its capacity in dependency modeling. The experimental results demonstrate that ANT achieves comparable performance with mainstream models in much higher efficiency and has great potential in various applications like latent interpolation and semi-supervised learning.

Submission Length: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=a6A1l3rsbU&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DTMLR%2FAuthors%23your-submissions)

Changes Since Last Submission: Fix the format problems.

Assigned Action Editor: ~Vincent_Dumoulin1

Submission Number: 5171