Redundancy Aware Multiple Reference Based Gainwise Evaluation of Extractive Summarization
Abstract: While very popular for evaluating extractive summarization task, the ROUGE metric has long been criticized for its lack of semantic awareness and its ignorance about the ranking quality of the summarizer. Thanks to previous research that has addressed these issues by proposing a gain-based automated metric called \textit{Sem-nCG}, which is both rank and semantic aware. However, \textit{Sem-nCG} does not consider the amount of redundancy present in a model-generated summary and currently does not support evaluation with multiple reference summaries. Unfortunately, addressing both these limitations simultaneously is not trivial. Therefore, in this paper, we propose a redundancy-aware \textit{Sem-nCG} metric and demonstrate how this new metric can be used to evaluate model summaries against multiple references. We also explore different ways of incorporating redundancy into the original metric through extensive experiments. Experimental results demonstrate that the new redundancy-aware metric exhibits a higher correlation with human judgments than the original \textit{Sem-nCG} metric for both single and multiple reference scenarios.
Submission Length: Regular submission (no more than 12 pages of main content)
Previous TMLR Submission Url: https://openreview.net/forum?id=0shW8mT1Z6&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DTMLR%2FAuthors%23your-submissions)
Changes Since Last Submission: The previous submission was desk-rejected because of font mismatch. The font has been revised to match the template.
Assigned Action Editor: ~Colin_Raffel1
Submission Number: 1553
Loading