NATURE: Natural Auxiliary Text Utterances forRealistic Spoken Language Evaluation

David Alfonso-Hermelo, Ahmad Rashid, Abbas Ghaddar, Philippe Langlais, Mehdi Rezagholizadeh

Keywords: slot filling, intent detection, dialog system, natural language understanding, virtual assistant, intent classification

TL;DR: We present a set of simple speech-oriented operators and show that simple pattern alterations deteriorate significantly the performance of state-of the-art models for virtual assistants.

Abstract: Slot-filling and intent detection are the backbone of conversational agents such as voice assistants and they are active areas of research. Even though state-of-the-art techniques on publicly available benchmarks show impressive performance, their ability to generalize to realistic scenarios has yet to be improved. In this work, we present NATURE, a set of simple spoken language oriented transformations, applied to the evaluation set of datasets, to introduce human spoken language variations while preserving the semantics of an utterance. We apply NATURE to common slot-filling and intent detection benchmarks and demonstrate that simple deviations from the standard test set by NATURE can deteriorate model's performance significantly. Additionally, we apply different strategies to mitigate the effects of NATURE and report that data-augmentation leads to some improvement.

URL: https://github.com/dahrs/sf_id_benchmarks