Classifiers are Better Experts for Controllable Text Generation

Anonymous

Keywords: text generation, controllable text generation, language model

TL;DR: Guiding language model with a free-form classifier for controllable text generation

Abstract:

This paper proposes a simple method for controllable text generation based on importance sampling, namely CAIF sampling. Using an arbitrary third-party text classifier, we adjust a small part of a language model's logits and guide text generation towards or away from classifier prediction.

We show that the proposed method significantly outperforms recent PPLM, GeDi, and DExperts on PPL and sentiment accuracy based on the external classifier of generated texts. A the same time, it is also easier to implement and tune, and has significantly fewer restrictions and requirements.