The C-Score–Proposing a Reading Comprehension Metrics as a Common Evaluation Measure for Text Simplification

Irina Temnikova, Galina Maneva

Abstract: This article addresses the lack of common approaches for text simplification evaluation, by presenting the first attempt for a common evaluation metrics. The article proposes reading comprehension evaluation as a method for evaluating the results of Text Simplification (TS). An experiment, as an example application of the evaluation method, as well as three formulae to quantify reading comprehension, are presented. The formulae produce a unique score, the C-score, which gives an estimation of the user’s reading comprehension of a certain text. The score can be used to evaluate the performance of a text simplification engine on pairs of complex and simplified texts or to compare the performances of different TS methods using the same texts. The approach can be particularly useful for the modern crowdsourcing approaches, such as those employing Amazon’s Mechanical Turk1 or CrowdFlower2. The aim of this paper is thus to propose an evaluation approach and to motivate the TS community to start a relevant discussion, in order to come up with a common evaluation metrics for this task.