Extractive-Abstractive: A Two-Stage Model for Long Text Summarization

Rui Liang, Jianguo Li, Li Huang, Ronghua Lin, Yu Lai, Dan Xiong

Abstract: Currently, the mainstream text summarization techniques are divided into extractive and abstractive methods. Extractive method is suitable for long texts with a clear structure, while abstractive method is suitable for short texts. In this paper, we aim to address the problems of missing key words and incomplete overview that are usually caused by abstractive method in the face of long texts. To solve this problem, we propose a two-stage model that uses both extractive and abstractive methods for generating summaries. Firstly, we use multi-layer BiLSTM for long text summary extraction. Secondly, we use the classical UniLM as the base model while adding a novel copy mechanism to tackle out-of-vocabulary (OOV) problem and using the sparse softmax to avoid overfitting. Extensive experiments demonstrate that our models perform better than other baseline models, and our models can generate higher quality summaries.