Discourse-Based Evaluation of Language Understanding

Damien Sileo, Tim Van-De-Cruys, Camille Pradel, Philippe Muller

Keywords: Natural Language Understanding, Pragmatics, Discourse, Semantics, Evaluation, BERT, Natural Language Processing

TL;DR: Semantics is not all you need

Abstract:

New models for natural language understanding have made unusual progress recently, leading to claims of universal text representations. However, current benchmarks are predominantly targeting semantic phenomena; we make the case that discourse and pragmatics need to take center stage in the evaluation of natural language understanding.
We introduce DiscEval, a new benchmark for the evaluation of natural language understanding, that unites 11 discourse-focused evaluation datasets.
DiscEval can be used as supplementary training data in a multi-task learning setup, and is publicly available, alongside the code for gathering and preprocessing the datasets.
Using our evaluation suite, we show that natural language inference, a widely used pretraining task, does not result in genuinely universal representations, which opens a new challenge for multi-task learning.

Code: https://github.com/disceval/DiscEval

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:1907.08672/code)