What Have We Achieved on Non-autoregressive Translation?

Anonymous

Abstract: Recent advances have made non-autoregressive (NAT) translation comparable to autoregressive methods (AT). However, their evaluation using BLEU has been shown to weakly correlate with human annotations. Limited research compares non-autoregressive translation and autoregressive translation comprehensively, leaving uncertainty about the true proximity of NAT to AT. To address this gap, we systematically evaluate four representative NAT methods across various dimensions, including human evaluation. Our empirical results demonstrate that despite narrowing the performance gap, state-of-the-art NAT still underperforms AT under more reliable evaluation metrics. Furthermore, we discover that explicitly modeling dependencies is crucial for generating natural language and generalizing to out-of-distribution sequences.

Paper Type: long

Research Area: Machine Translation

Contribution Types: Model analysis & interpretability

Languages Studied: English; German; Romanian; Chinese

Preprint Status: We are considering releasing a non-anonymous preprint in the next two months (i.e., during the reviewing process).

A1: yes

A1 Elaboration For Yes Or No: Section. Limitaitons

A2: n/a

A3: yes

A3 Elaboration For Yes Or No: Section Abstract & 1 Introduction

B: yes

B1: yes

B1 Elaboration For Yes Or No: Section 3 Experiment and Setup & Appendix C

B2: n/a

B3: yes

B3 Elaboration For Yes Or No: Section Ethics Consideration

B4: yes

B4 Elaboration For Yes Or No: Section Ethics Consideration

B5: yes

B5 Elaboration For Yes Or No: Section 3 Experiment and Setup & Appendix C

B6: yes

B6 Elaboration For Yes Or No: Section 3 Experiment and Setup & Appendix C

C: yes

C1: yes

C1 Elaboration For Yes Or No: Section 3 Experiment and Setup & Appendix C

C2: yes

C2 Elaboration For Yes Or No: Section 3 Experiment and Setup & Appendix C

C3: yes

C3 Elaboration For Yes Or No: Section 4 Translation Quality

C4: yes

C4 Elaboration For Yes Or No: Section 3 Experiment and Setup & Appendix C

D: yes

D1: yes

D1 Elaboration For Yes Or No: Appendix D

D2: yes

D2 Elaboration For Yes Or No: Appendix D

D3: yes

D3 Elaboration For Yes Or No: Appendix D & Section Ethics Consideration

D4: yes

D4 Elaboration For Yes Or No: Appendix D & Section Ethics Consideration

D5: yes

D5 Elaboration For Yes Or No: Appendix D & Section Ethics Consideration

E: no

E1: n/a