Coherent or Not? Stressing a Neural Language Model for Discourse Coherence in Multiple Languages

Dominique Brunato, Felice Dell'Orletta, Irene Dini, Andrea Amelio Ravelli

Abstract: In this study, we investigate the capability of a Neural Language Model (NLM) to distinguish between coherent and incoherent text, where the latter has been artificially created to gradually undermine local coherence within text. While previous research on coherence assessment using NLMs has primarily focused on English, we extend our investigation to multiple languages. We employ a consistent evaluation framework to compare the performance of monolingual and multilingual models in both in-domain and out-domain settings. Additionally, we explore the model’s performance in a cross-language scenario.