Improve the Diversity and Novelty for Open-Ended Neural Text Generation via Inverse Probability Weighting

Xinran Zhang, Maosong Sun, Jiafeng Liu, Xiaobing Li

Abstract: Stochastic sampling methods for open-ended neural text generation severely affect the diversity and novelty of generated texts. Traditional sampling methods only prune the low-likelihood part of the predicted distribution to trade-off between the fluency and diversity of generated texts. They do not manipulate the high-likelihood part, which leads to the likelihood trap that induces low diversity and boredom. They also do not directly leverage that human does not always favor high-likelihood texts. Inspired by these, we propose a novel sampling method that rescales the high-likelihood part of the distribution with inverse probability weighting. It increases the diversity and novelty of generated texts by rescaling and penalizing the high-likelihood part of the predicted distribution, and preserves the fluency by using multi-filtering truncation on the low-likelihood part. Experimental results show that compared with traditional methods our algorithm can significantly increase the diversity and novelty of generated texts without sacrificing fluency.