NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language EvaluationDownload PDF

Published: 11 Oct 2021, Last Modified: 23 May 2023NeurIPS 2021 Datasets and Benchmarks Track (Round 2)Readers: Everyone
Keywords: slot filling, intent detection, voice assistant, dialog system, natural language understanding, virtual assistant, intent classification
TL;DR: We present a set of simple speech-oriented operators and show that simple pattern alterations deteriorate significantly the performance of state-of the-art models for Voice Assistants.
Abstract: Slot-filling and intent detection are the backbone of conversational agents such as voice assistants, and are active areas of research. Even though state-of-the-art techniques on publicly available benchmarks show impressive performance, their ability to generalize to realistic scenarios is yet to be demonstrated. In this work, we present NATURE, a set of simple spoken-language-oriented transformations, applied to the evaluation set of datasets, to introduce human spoken language variations while preserving the semantics of an utterance. We apply NATURE to common slot-filling and intent detection benchmarks and demonstrate that simple perturbations from the standard evaluation set by NATURE can deteriorate model performance significantly. Through our experiments we demonstrate that when NATURE operators are applied to evaluation set of popular benchmarks the model accuracy can drop by up to 40%.
Supplementary Material: zip
Contribution Process Agreement: Yes
Dataset Url:
Author Statement: Yes
8 Replies