Comparison of diverse decoding methods from conditional language models

Reno Kriz, Daphne Ippolito, João Sedoc, Maria Kustikova, Chris Callison-Burch

Abstract: While conditional language models have greatly improved in their ability to output high-quality natural language, many NLP ap- plications benefit from being able to generate a diverse set of candidate sequences. Diverse decoding strategies aim to, within a given- sized candidate list, cover as much of the space of high-quality outputs as possible, leading to improvements for tasks that re-rank and com- bine candidate outputs. Standard decoding methods, such as beam search, optimize for generating high likelihood sequences rather than diverse ones, though recent work has fo- cused on increasing diversity in these meth- ods. In this work, we perform an extensive survey of decoding-time strategies for generat- ing diverse outputs from conditional language models. We also show how diversity can be improved without sacrificing quality by over- sampling additional candidates, then filtering to the desired number.