NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language EvaluationDownload PDF

19 Aug 2021, 21:10 (edited 14 Jan 2022)NeurIPS 2021 Datasets and Benchmarks Track (Round 2)Readers: Everyone
  • Keywords: slot filling, intent detection, voice assistant, dialog system, natural language understanding, virtual assistant, intent classification
  • TL;DR: We present a set of simple speech-oriented operators and show that simple pattern alterations deteriorate significantly the performance of state-of the-art models for Voice Assistants.
  • Abstract: Slot-filling and intent detection are the backbone of conversational agents such as voice assistants, and are active areas of research. Even though state-of-the-art techniques on publicly available benchmarks show impressive performance, their ability to generalize to realistic scenarios is yet to be demonstrated. In this work, we present NATURE, a set of simple spoken-language-oriented transformations, applied to the evaluation set of datasets, to introduce human spoken language variations while preserving the semantics of an utterance. We apply NATURE to common slot-filling and intent detection benchmarks and demonstrate that simple perturbations from the standard evaluation set by NATURE can deteriorate model performance significantly. Through our experiments we demonstrate that when NATURE operators are applied to evaluation set of popular benchmarks the model accuracy can drop by up to 40%.
  • Supplementary Material: zip
  • URL: https://github.com/rali-udem/sf_id_benchmarks
  • Contribution Process Agreement: Yes
  • Dataset Url: https://github.com/rali-udem/sf_id_benchmarks
  • Author Statement: Yes
8 Replies

Loading