Lightweight Reference-Less Summary Quality Evaluation via Key Feature Extraction

Shunan Zang, Chuang Zhang, Jingwen Lin, Xiaojun Chen, Shuai Zhang

Abstract: One of the main problems with automatic text summarization is the lack of a “gold standard" for summary quality evaluation. ROUGE [9] is the most widely used evaluation metric for summary quality. However, its evaluation merely concentrates on reference summary and overlap features of sentences rather than focusing on more critical semantic features. Some other exiting methods have issues with improper noise handling and high cost. To solve these problems, we propose a lightweight reference-less summary quality evaluation method (SE-tiny), which evaluates the summary from two aspects: the summary’s self-quality and the degree of matching the features of the summary with the key features of the source text. Then, we optimize computational efficiency and space cost. Compared with existing methods, SE-tiny improves the quality of evaluation and reduces the cost. Besides, our method does not rely on reference summaries and can be generalized to evaluation on summarization datasets. For the goal of reproducibility, we make the SE-tiny project’s code and models available.