PPLM Revisited: Steering and Beaming a Lumbering Mammoth to Control Text Generation

Anonymous

Keywords: natural language generation, language models, pplm, reproducibility, generalization

Abstract:

Dathathri et al., 2020 introduced the Plug and Play Language Model (PPLM) as a configurable method for controlled language generation. Since then, the publication has been cited more than 200 times and the official implementation received >800 stars on GitHub. Such popularity can be explained by the fact that PPLM achieves fine-grained control of attributes such as topics and sentiment while retaining fluency without retraining the language model (LM).

In a blogpost accompanying the original paper, the authors compared any LM with an “unguided mammoth” whilst PPLM was presented as a mouse sitting on top of the mammoth who steers it in the right direction. However, the interplay between the prompt and the BoW is underexplored in the paper and the original blog post. Moreover, to the best of our knowledge, the impact of the interplay between the prompt and the BoW on the controllability of the text has also not yet been described elsewhere.

In this blog post, we address both of the issues as well as the issue of the reproducibility and sensitivity to hyperparameter settings of PPLM. We aim to provide more insights into the workings of PPLM and, to some extent, into Natural Language Generation (NLG) controllability in general.

Based on the results of the experiments we conclude that producing topic related texts is difficult with the hyperparameters provided in the original paper, but pushing the distribution of the generated words more towards the targeted subject leads to enhanced results. On top of that, we also conclude that carefully picking the words in the BoW is important to control the PPLM, and modifying the weight might help control the PPLM to generate the texts that are relevant to the exact words that we like.

ICLR Paper: https://openreview.net/pdf?id=H1edEyBKDS