Fact-based content weighting for evaluating abstractive summarisation

Xinnuo Xu, Ondrej Dusek, Verena Rieser, Ioannis Konstas

Abstract: Abstractive summarisation is notoriously hard to evaluate since standard word-overlap-based metrics are insufficient. We introduce a new evaluation metric which is based on fact-level content weighting, ie relating the facts of the document to the facts of the summary. We fol-low the assumption that a good summary will reflect all relevant facts, ie the ones present in the ground truth (human-generated refer-ence summary). We confirm this hypothe-sis by showing that our weightings are highly correlated to human perception and compare favourably to the recent manual highlight-based metric of Hardy et al.(2019).