Goal-Directed Story Generation: Augmenting Generative LanguageModels with Reinforcement Learning

Anonymous

Paper Link: https://openreview.net/forum?id=SZQGW30wS3n

Paper Type: Long paper (up to eight pages of content + unlimited references and appendices)

Abstract:

The advent of large pre-trained generative language models has provided a common framework for AI story generation via sampling the model to create sequences that continue the story.
However, sampling alone is insufficient for story generation.
In particular, it is hard to direct a language model to create stories to reach a specific goal event.
We present two automated techniques grounded in deep reinforcement learning and reward shaping to control the plot of computer-generated stories.
The first utilizes proximal policy optimization to fine-tune an existing transformer-based language model to generate text continuations and be goal-seeking.
The second extracts a knowledge graph from the unfolding story, which a policy network uses with graph attention to select a candidate continuation generated by a language model.
We report on automated metrics on how often stories achieve a given goal event and human participant rankings of coherence and overall story quality compared to baselines and ablations.