Content Richness Evaluation Method For Abstractive Summarization

Xingyue Zhang, Dingxin Hu, Baofeng Li, Yu Qin, Lei Li

Abstract: The abstractive summarization model can generate summaries with factual errors, providing error correction models can solve this general problem. Deleting erroneous facts is the most commonly used important strategy in the error correction process. However, we found through experiments that when using this strategy, the error correction model is more inclined to delete all suspicious content, resulting in a significant reduction in information volume. To ensure both the information richness and factual consistency of summaries, this paper proposes a content richness evaluation method named ASCRE. It evaluates the richness of a summary by measuring the quantity and effectiveness of the information contained within it. This paper demonstrates through experiments that ASCRE can effectively evaluate the quality of abstractive summarization system and error correction models. At the same time, we propose a factual consistency and content richness trade off line as the quality evaluation benchmark.