Unlocking the Power of GANs in Non-Autoregressive Text Generation under Weak Conditions

TMLR Paper5171 Authors

Abstract: Non-autoregressive (NAR) models once attracted significant attention from the research community but have received considerably less focus in the pursuit of general artificial intelligence. Our analysis reveals that the convergence problem in existing fully NAR models trained under Maximum Likelihood Estimation (MLE) becomes more severe in tasks where the input does not provide definitive semantic constraints for the output. We denote these input conditions as weak conditions, which encompass most ``creative'' tasks. Consequently, existing fully NAR models struggle to achieve satisfactory performance in such scenarios and remain confined to limited application domains. This limitation hinders fully NAR models from keeping pace with the rapidly evolving demands of diverse and challenging tasks. Unlike MLE, which is fundamentally incompatible with NAR models, Generative Adversarial Networks (GANs) offer superior theoretical convergence guarantees and inference characteristics for fully NAR architectures. We therefore propose an Adversarial Non-autoregressive Transformer (ANT) based on GANs specifically designed for weak condition tasks. ANT incorporates two key innovations: 1) Position-Aware Self-Modulation to provide more effective input signals, and 2) Dependency Feed Forward Network to enhance dependency modeling capabilities. Experimental results demonstrate that ANT achieves comparable performance to mainstream models while maintaining significantly higher efficiency, and exhibits substantial potential in various applications including latent interpolation and semi-supervised learning.

Submission Length: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=a6A1l3rsbU&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DTMLR%2FAuthors%23your-submissions)

Changes Since Last Submission: Revise the papers based on the comments from reviewers. Detailed changes are illustrated in the reply.

Assigned Action Editor: ~Vincent_Dumoulin1

Submission Number: 5171