Structure Controllable Text Generation

Liming DENG, Long WANG, Binzhu WANG, Jiang Qian, Bojin Zhuang, Shaojun Wang, Jing Xiao

Keywords: Natural language generation, structure representation, structure controlling, conditional language model, structure aware transformer

Abstract: Controlling the presented forms (or structures) of generated text are as important as controlling the generated contents during neural text generation. It helps to reduce the uncertainty and improve the interpretability of generated text. However, the structures and contents are entangled together and realized simultaneously during text generation, which is challenging for the structure controlling. In this paper, we propose an efficient, straightforward generation framework to control the structure of generated text. A structure-aware transformer (SAT) is proposed to explicitly incorporate multiple types of multi-granularity structure information to guide the text generation with corresponding structure. The structure information is extracted from given sequence template by auxiliary model, and the type of structure for the given template can be learned, represented and imitated. Extensive experiments have been conducted on both Chinese lyrics corpus and English Penn Treebank dataset. Both automatic evaluation metrics and human judgement demonstrate the superior capability of our model in controlling the structure of generated text, and the quality ( like Fluency and Meaningfulness) of the generated text is even better than the state-of-the-arts model.

One-sentence Summary: A straightforward, interpretable structure controlling text generation framework is proposed, which is capable of learning and controlling multigranularity sequence structure from character-level to sentence-level structure.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=4rmZ6aXk1r