An Exploitation of Heterogeneous Graph Neural Network for Extractive Long Document Summarization

Anonymous

Abstract: Heterogeneous Graph Neural Networks (HeterGNN) has been recently introduced as an emergent approach for many Natural Language Processing (NLP) tasks by enriching the complex information between word and sentence. In this paper, we try to improve the performance of Extractive Document Summarization (EDS) for long-form documents based on the concept of HeterGNN. Specifically, long documents (e.g., Scientific Papers) are truncated for most neural-based models, which leads to the challenge in terms of information loss of inter-sentence relations. In this regard, we present a new method by exploiting the capabilities of HeterGNN and pre-trained language models. Particularly, BERT is considered for improving the sentence information into the Heterogenous graph layer. Accordingly, two versions of the proposed method are presented which are: i) Multi Graph Neural Network (MTGNN-SUM), by combining both heterogeneous graph layer and graph attention layer; and ii) HeterGNN with BERT (HeterGNN-BERT-SUM), by integrating BERT directly into the heterogeneous graph structure. Experiments on two benchmark datasets of long documents such as PubMed and ArXiv show that our method outperforms state-of-the-art models in this research field

Paper Type: long