Sentop: sentence-level prefix prompt for controllable abstractive summarization

Shu Zhao, Yang Sun, Jie Chen, Zhen Duan, Chunhui Zou, Yuanfang Cheng, Fugui He

Abstract: ive text summarization aims to paraphrase the given corpus and generate new sentences while retaining key information. In recent works, abstractive summarization task has been commonly modeled using Transformers, which are fine-tuned by maximizing the likelihood of the output text sequences. However, such methods face two challenges. First, due to the difference between training and evaluation stages, the model uses ground truth tokens to predict the next token during training, while it relies on its own previous predictions during inference, leading to exposure bias. Second, traditional fine-tuning adjusts all parameters of the pre-trained language model which requires a huge amount of computation. To this end, we propose SentoP, a method for controllable abstractive text summarization using sentence-level prefix prompts. Our method trains a small prefix network on top of a frozen Seq2Seq Transformer and uses ranking loss based on contrastive learning method to align sentence-level semantics for controlled summary generation. Compared with traditional fine-tuning, our method which tunes only 20% of the parameters, achieves over 10% consistent improvement on average in few-shot settings and also obtains comparable performance in the full data settings.

External IDs:dblp:journals/mlc/ZhaoSCDZCH25