Quantifying Exposure Bias for Neural Language Generation

Tianxing He, Jingzhao Zhang, Zhiming Zhou, James Glass

TL;DR: We show that exposure bias could be much less serious than it is currently assumed to be for MLE LM training.

Abstract: The exposure bias problem refers to the training-inference discrepancy caused by teacher forcing in maximum likelihood estimation (MLE) training for auto-regressive neural network language models (LM). It has been regarded as a central problem for natural language generation (NLG) model training. Although a lot of algorithms have been proposed to avoid teacher forcing and therefore to alleviate exposure bias, there is little work showing how serious the exposure bias problem is. In this work, we first identify the auto-recovery ability of MLE-trained LM, which casts doubt on the seriousness of exposure bias. We then develop a precise, quantifiable definition for exposure bias. However, according to our measurements in controlled experiments, there's only around 3% performance gain when the training-inference discrepancy is completely removed. Our results suggest the exposure bias problem could be much less serious than it is currently assumed to be.

Keywords: language model, exposure bias, language generation

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1905.10617/code)