Discourse-Aware Prompt Design for Text Generation

Anonymous

Abstract: Current efficient fine-tuning methods (e.g., adapters, prefix-tuning, etc.) have optimized conditional text generation via training a small set of extra parameters of the neural language model, while freezing the rest for efficiency. While showing strong performance on some generation tasks, they don't generalize across all generation tasks. In this work, we show that prompt based conditional text generation can be improved with simple and efficient methods that simulate modeling the discourse structure of human written text.We introduce two key design choices: First, we show that a higher-level discourse structure of human written text can be modelled with hierarchical blocking on prefix parameters. It enables spanning different parts of the input and output text and yields more coherent output generations. Second, we propose sparse prefix tuning by introducing attention sparsity on the prefix parameters at different layers of the network and learn sparse transformations on the softmax-function, respectively. We find that sparse attention enables the prefix-tuning to better control of the input contents (salient facts) yielding more efficient tuning of the prefix-parameters. Our experiments show that structured design of prefix parameters can yield more coherent, faithful and relevant generations than baseline prefix-tuning on all generation tasks and perform at par with fine-tuning while being more efficient.

Paper Type: long