An Improved Template Representation-based Transformer for Abstractive Text Summarization

Jiaming Sun, Yunli Wang, Zhoujun Li

Abstract: Text summarization plays an important role in various NLP applications. Using templates with generation methods is an effective way to address abstractive summarization. However, existing template-enhanced generation approaches use templates in a naive way and mainly adopt RNN-based Seq2Seq models, so they cannot make full use of valid information in the templates and suffer from templates' noise. To mitigate these problems, we propose a new abstractive summarization model called Summarization Transformer with Template-aware Representation (STTR), which uses a template-aware document encoding module and a document representation shifting loss to preserve the useful information and filter the noise of the template. The experiments on the Gigaword and LCSTS datasets show that our method outperforms baseline models and achieves a new state-of-the-art.