Eye Tracking Based Cognitive Evaluation of Automatic Readability Assessment Measures

Keren Gruteke Klein, Shachar Frenkel, Omer Shubi, Yevgeni Berzak

Keywords: eye tracking, readability, text simplification

TL;DR: Eye Tracking Based Cognitive Evaluation of Automatic Readability Assessment Measures

Abstract: Automatic Readability Assessment (ARA) is widely used in many real-world scenarios. Over the past century, ARA measures have primarily been developed and evaluated on reading comprehension outcomes and on human annotations of text readability levels. In this work, we propose an alternative, eye tracking-based cognitive framework which directly taps into a key aspect of readability that has been largely overlooked in ARA research: reading ease. We use this framework for evaluating a broad range of traditional and state-of-the-art readability measures, including two systems widely used in education, by quantifying their ability to account for reading facilitation effects in text simplification, as well as text reading ease more broadly. Our analyses suggest that existing readability measures are poor predictors of reading facilitation and reading ease, outperformed by word properties commonly used in psycholinguistics, and in particular by surprisal.

Submission Number: 15