Self-Improvement of Non-autoregressive Model via Sequence-Level Distillation

Yusheng Liao, Shuyang Jiang, Yiqi Li, Yu Wang, Yanfeng Wang

Submission Type: Regular Long Paper

Submission Track: Machine Translation

Submission Track 2: Natural Language Generation

Keywords: non-autoregressive transformers, self-improvement, distillation

TL;DR: A simple yet effective method to generate the distilled data which is more suitable for non-autoregressive model to learn.

Abstract: Although Non-autoregressive Transformer (NAT) models have achieved great success in terms of fast inference speed, this speedup comes with a performance drop due to the inherent \emph{multi-modality} problem of the NAT model. Previous works commonly alleviate this problem by replacing the target side of the raw data with distilled data generated by Autoregressive Transformer (AT) models. However, the multi-modality problem in the distilled data is still significant and thus limits further improvement of the NAT models. In this paper, we propose a method called Sequence-Level Self-Distillation (SLSD), which aims to generate distilled data by the NAT model itself, eliminating the need for additional teacher networks. Furthermore, SLSD can adapt to different NAT models without precise adjustments since the self-distilled data is generated from the same types of NAT models. We conduct extensive experiments on WMT14 EN$\leftrightarrow$DE and WMT16 EN$\leftrightarrow$RO and choose four classic NAT models as the backbones to validate the generality and effectiveness of SLSD. The results show that our approach can consistently improve all models on both raw data and distilled data without sacrificing the inference speed.

Submission Number: 541