Bipartite Graph Pre-training for Unsupervised Extractive Summarization with Graph Convolutional Auto-Encoders

Qianren Mao, Shaobo Zhao, Jiarui Li, Xiaolei Gu, Shizhu He, Bo Li, Jianxin Li

Submission Type: Regular Long Paper

Submission Track: Summarization

Submission Track 2: Machine Learning for NLP

Keywords: Bipartite Graph, Graph Pre-training, Unsupervised Extractive Summarization, Graph Convolutional Auto-Encoders

TL;DR: Our proposed graph pre-training auto-encoder models intra-sentential features and inter-sentential features through sentence-word bipartite graphs, enabling extract significant sentences for unsupervised summarization.

Abstract:

Pre-trained sentence representations are crucial for identifying significant sentences in unsupervised document extractive summarization.
However, the traditional two-step paradigm of pre-training and sentence-ranking, creates a gap due to differing optimization objectives.
To address this issue, we argue that utilizing pre-trained embeddings derived from a process specifically designed to optimize informative and distinctive sentence representations helps rank significant sentences.
To do so, we propose a novel graph pre-training auto-encoder to obtain sentence embeddings by explicitly modelling intra-sentential distinctive features and inter-sentential cohesive features through sentence-word bipartite graphs.
These fine-tuned sentence embeddings are then utilized in a graph-based ranking algorithm for unsupervised summarization.
Our method is a plug-and-play pre-trained model that produces predominant performance for unsupervised summarization frameworks by providing summary-worthy sentence representations.
It surpasses heavy BERT- or RoBERTa-based sentence representations in downstream tasks.

Submission Number: 3818