HierMDS: a hierarchical multi-document summarization model with global-local document dependencies

Shuimin Li, Jungang Xu

Abstract: Multi-document summarization (MDS) has attracted increasing attention in recent years. Most existing MDS systems simply encode the flat connected sequence of multiple documents, which limits the representation capabilities for multi-documents. To address this issue, we propose a Hierarchical Multi-Document Summarization Model with Global-Local Document Dependencies (HierMDS). HierMDS consists of five sub-blocks, i.e., an embedding block, an internal document encoding block, a local document encoding block, a global document encoding block, and a fusion block, which are stacked in a hierarchical structure to gradually produce dependency-enriched document representations. Specifically, the embedding block encodes tokens, and the internal document encoding block encodes each document. Then, for a certain document, two kinds of document dependencies are extracted: (1) The global document dependency indicates that the representation of this document is affected by all the other documents. (2) The local document dependency indicates that the representation of this document is only affected by the relevant documents. We suppose that the global document dependency represents the global background information, while the local document dependency condenses the most relevant information. To be specific, the global document encoding block modeled with the vanilla transformer layer encodes the global document dependencies, and the local document encoding block modeled with the graph attention neural networks encodes the local document dependencies. Finally, HierMDS produces document dependency-enriched representations by fusing the local and global document dependencies with the fusion block. Experimental results on Multi-News and DUC-2004 datasets have demonstrated competitive advantages of HierMDS compared with several state-of-the-art MDS models.